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Preface 



In 2002, Tennessee passed its first Public Charter School legislation. Four 
schools were successful in gaining approval to begin operating in the 2003-2004 
academic year. The second cohort of three charter schools began operation during the 
2004-2005 academic year. According to the charter school law, these charter schools 
were granted “maximum flexibility” to achieve alternative ways for public schools to 
educate school children. Though the flexibility granted to these schools is considered an 
advantage, previous research on charter schools has demonstrated mixed results in 
their success. 

The purpose of the present evaluation study was to examine the progress made 
in program implementation, school climate, and student achievement by the seven 
charter schools. A “mixed-methods” design, encompassing both qualitative and 
quantitative data, was employed. The questions upon which the evaluation methods are 
based relate to the progress of individual schools and the overall group in implementing 
desired strategies for curriculum, instruction, and organization, and in attaining the goals 
of No Child Left Behind by bringing every child to proficiency in reading and 
mathematics on the TCAP by 2014. It should be noted that student achievement will be 
more fully examined in a supplemental report when student-level data become 
available. 

The work was conducted by the Center for Research in Educational Policy 
(CREP), a state of Tennessee Center of Excellence, located at The University of 
Memphis. 
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Executive Summary 

The purpose of the present evaluation study was to examine the progress made 
in program implementation, school climate, and student achievement by the seven TN 
charter schools. Four of these charter schools opened in the 2003-04 school year and 
an additional three schools opened in the 2004-05 school year. Seven evaluation 
questions guided the methodology for this study. Student achievement results 
(Question 1) and start-up experiences (Question 7) are addressed in separate reports. 
The following evaluation questions (2-6) are addressed in this report. 

2. What is the frequency of usage of various traditional and alternative (student- 
centered) instructional strategies in the charter schools and compared to national 
norms? 

3. What is the school climate at the charter schools and how does the climate 
compare to national norms? 

4. To what degree and levels of quality are the goals and strategies of the charter 
school being implemented? 

5. What are teacher reactions to and experiences in the charter school? What are 
the adequacy and quality of professional development and resources? 

6. What are parent (caregiver) reactions to and experiences with the charter 
school? 
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Description of the Charter Schools 

Circles of Success Learning Academy (COSLA) 

COSLA is an urban school located in Memphis, Tennessee. In its second year of 
operation, COSLA enrolled a total of 72 students in grades K-3. All are African 
American, and nearly all (91 .6%) were eligible for free or reduced price lunch. As a 
literacy-focused school using the Success for All reading program, COSLA emphasizes 
the integration of reading and writing activities across all curricular areas. The school 
uses scientifically-based methods and strategies in the delivery of the curriculum. 
Memphis Academy of Health Sciences (MAHS) 

MAHS is located in a high-poverty area north of downtown Memphis, Tennessee. 
The school served 180 sixth and seventh grade students in its second year of operation. 
African American students represented 100% of the school’s population; 75% of the 
students received free or reduced-price lunches. The curriculum is a standards-based, 
interdisciplinary program that incorporates interdisciplinary projects and experiential 
learning centered on a health science theme. 

Memphis Academy of Science and Engineering (MASE) 

MASE is housed in an office building in downtown Memphis. During the 2004- 
2005 school year, there were 251 seventh and eighth grade students. The students 
were predominantly African American (97%), and 68% qualified for free or reduced- 
price lunch. The program includes longer days, Saturday school, high technology use, 
continuous monitoring of student progress, and an intense focus on core curricula 
integrated across subjects. 
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Smithson-Craighead Academy (SC A) 

This urban charter school, located in the city of Nashville, Tennessee, is in its 
second year of operation. It served 148 students in kindergarten though fourth grade. 
The students were predominantly African American (99%), and all were eligible for free 
or reduced price lunch. The mission of the school is to meet the academic and social 
needs of at-risk children through mastery of basic academic skills, student self control, 
obedience, and diligence. 

City University School of Liberal Arts (CityU) 

Cityll is located in a Baptist church in an urban area of Memphis, Tennessee. 
The school enrolled 121 ninth grade students in its first year of operation. Just over 
98% of the students were African American, and 78% were eligible for free or reduced 
lunch. The curriculum developed and employed at CityU has a liberal arts core and is 
aligned with College Board standards to build a strong college-oriented focus. 

Star Academy 

Star Academy is located in the outskirts of Memphis, Tennessee. In its first year 
of operation the school served 116 students in kindergarten through second grade. The 
student population was predominantly African American, with approximately 90 percent 
qualifying for free or reduced price lunch. Reading and language arts were the primary 
focus for the first year. The McGraw-Hill series provides the core curricular materials. 
Yo! Academy 

Yo! Academy of Visual and Performing Arts is located in southwest Memphis in 
an industrial warehouse site. During the first year of operation, the school served 131 
students in the tenth through twelfth grades. All students were African American and 
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were eligible for free or reduced price meals. The mission of the school is to provide at- 
risk students with a safe environment to pursue rigorous academic preparation while 
incorporating the visual and performing arts. 

Method 

A mixed methods design, encompassing both qualitative and quantitative data, 
was employed. The instruments used in the data collection were: (1) School 
Observation Measure (SOM®), (2) Rubric for Student Centered Activities (RSCA®), (3) 
School Climate Inventory (SCI®), (4) principal interview, (5) teacher focus group, (6) 
student focus group, (7) Charter School Teacher Questionnaire, (8) Charter School 
Parent Questionnaire, and (9) Program Implementation Benchmarking. 

A “site researcher” from CREP was assigned major data collection responsibility 
for each charter school. This individual and support research staff visited the assigned 
school several times during the year to conduct the observation visits, administer the 
questionnaires, and conduct the interview and focus groups. In addition, the site 
researcher worked with the school leadership team to develop “implementation 
benchmarks” describing beginning, intermediate, and full implementation phases and 
associated evidence indicators. At the end of the year, the site researcher met again 
with the leadership team to determine and identify the phase that had been achieved for 
each benchmark. Data from all instruments were then used to prepare individual school 
formative evaluation reports indicating status and progress during the year. 
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Results 



Question 2 1 . What is the frequency of usage of various traditional and alternative 
(student-centered) instructional strategies in the charter schools and compared 
to national norms? 

Direct instruction was the predominant instructional orientation in all charter 
schools. These rates were typically higher or comparable to rates observed in the 
national samples. Four of the schools (COSLA, MASE, SCA, and Star) employed 
student-centered instruction to a moderate degree. The most frequent types of student- 
centered instruction included cooperative/collaborative learning and the teacher acting 
as coach facilitator. All but one school (MAHS) implemented higher-order strategies at 
moderate levels. Observations at the three schools in their first year of operation 
(Cityll, Star, and Yo!) and COSLA revealed infrequent technology use. In contrast, 
technology use increased from the previous year n three of the first cohort of charter 
schools (MAHS, MASE, SCA). In some schools, the observed frequencies of student- 
centered strategies and technology usage were comparable to national norms, but the 
level of application was typically limited. 

Question 3. What is the school climate at the charter schools and how does the 
climate compare to national norms? 

School climate was a clear strength of all charter schools excepting SCA. SCA’s 
less positive (but still moderate) rating was due to low quantitative and qualitative 
responses on the Order dimension, also relative weakness in comparison with other 
schools. With regard to trends across time, three of the four Cohort I schools showed 

1 Questions 1 and 7 are addressed in other reports. 
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decreases in SCI ratings. However, the overall mean ratings were still high in both an 
absolute (above a 4.00) and relative sense (i.e., compared to national norms). 

Question 4. To what degree and levels of quality are the goals and strategies of 
the charter school being implemented? 

Not unexpectedly, the levels of benchmark implementation were more advanced 
among schools in their second year of operation compared to schools in their first year. 
However, the levels of implementation across most indicators were either the first or 
second phase of development across schools and years. Most schools were rated as 
moderate on their implementation of goals and strategies in the category of curriculum. 
Although some school leadership teams considered themselves to have made more 
progress in the implementation of benchmarks related to instruction, SOM and RSCA 
data did not consistently support this classification. Most charter schools were rated to 
be weaker in the implementation of support goals. The achievement of organizational 
goals was evaluated to be moderate in all but one school (Yol), which achieved less 
progress in this area. There was more variation across schools in the implementation of 
goals and strategies related to evaluation. The need to improve benchmarks 
contributed to the weaker ratings among schools in this area. 

Question 5. What are teacher reactions to and experiences in the charter school? 
What are the adequacy and quality of professional development and resources? 

Teachers’ reactions were very positive with respect to the school mission or 
educational program, but teachers were less enthusiastic about available resources and 
support. Among first cohort schools, teacher ratings were lower in the areas of 
resources and support than they had been in the previous year. Responses in all but 
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one school (Star) suggested deficiencies in meeting the requirements of special needs 
students. In general, teachers were less likely to agree that they received adequate 
assistance from state and local educational agencies, and in some cases external 
partners. There was more variation in the findings on the adequacy or quality of 
professional support. It was found to be strong in two schools (COSLA and Star) but 
weak in another (Yol). Teacher responses to professional development in the 
remaining schools were more moderate or mixed. 

Question 6. What are parent (caregiver) reactions to and experiences with the 
charter school? 

Parent satisfaction was quite strong in all the charter schools. The only areas for 
which they expressed less satisfaction were those related to the physical building, 
facilities, surroundings, transportation, and the lack of extracurricular activities or 
additional programs. Ratings in the category of involvement were more varied. In some 
schools, parent involvement was considered to be parent support or communication 
rather than active, consistent participation in the educational program and more 
academically-related activities. Surprisingly, the level and extent of parent participation 
did not seem to differ depending on whether the school was in its first or second year of 
operation. 

Recommendations 

Based on the overall findings, the following recommendations apply to the 
charter schools as a group and not necessarily to individual schools. The 
recommendations specific to each school can be found in the individual reports 
provided to each school. 
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First, charter schools could more closely align instructional practices with their 
school improvement plans and incorporate more research-based practices linked with 
student achievement outcomes. Many of the research-based practices call for the 
implementation of more student-centered strategies that entail higher-order thinking and 
technology integration. 

Second, order and discipline could be improved at most schools. School 
leadership teams may want to re-evaluate policies related to student conduct, discipline 
procedures, and attendance policies. Benchmark indicators targeting order might be 
developed to better formalize strategies to improve this dimension of school climate. 

Third, benchmark documents need to be modified to better align with objective 
indicators and available data. Planning, evaluation, and ultimately improvement can be 
facilitated by a sound, ambitious, and long-range benchmark document. 

Fourth, schools should make improvements in meeting the requirements of 
special needs students. Additional resources might be earmarked to hire extra staff and 
faculty with expertise in special education. Policies and procedures related to 
addressing these needs might be developed or reviewed. 

Efforts to continue developing supportive, collaborative relationships with the 
school district and external partners should continue. Strategies include having more 
open channels of communication and meetings to clarify policies, procedures, and 
available resources. Furthermore, representatives of charter schools should continue to 
seek the support and assistance of various external partners. 

The final recommendation is to increase active parent involvement or 
participation. Schools that had the highest levels of parent involvement also had formal 
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contracts or agreements with parents specifying the minimum number of hours of 
participation required. Benchmark documents that identify more specific strategies 
used to encourage involvement might be coupled with more objective evidence to 
indicate their effectiveness. 
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Second-Year Evaluation of Tennessee Charter Schools (2004 - 2005) 

This report presents results from the evaluation by the Center for Research in 
Educational Policy of the seven charter schools currently operating in the State of 
Tennessee. The first cohort of four schools began operation in the 2003-04 academic 
year and have completed their second full year of operation. The second cohort of 
three schools completed their initial academic year that began in 2004. By way of 
historical background, the passage of the first charter school law and establishment of 
the first charter school occurred in Minnesota in 1991. As of January 2004, 2,996 
charter schools were operating in the United States (Center for Education Reform, 
2004). Additionally, under the recent No Child Left Behind legislation, children who 
attend schools identified as needing improvement have the opportunity to enroll in 
charter schools located within their district (U.S. Department of Education, 2001). 
Although charter schools have been developed for a wide variety of reasons by diverse 
groups, they can be broadly characterized as publicly-funded schools that students can 
choose to attend. Additionally, they often operate with more autonomy than regular 
public schools, under a charter (contract) held by a public entity (e.g., school district, 
university, state department of education). The schools are also typically responsible 
for attaining, or making demonstrable progress toward, the goals set forth in their 
charter within a set amount of time. 

In 2002, the state of Tennessee passed its first public charter school legislation. 
According to this law, charter schools were intended to receive “maximum flexibility” to 
achieve alternative ways for public schools to educate school children. Though the 
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flexibility granted to these schools is considered an advantage, previous research on 
charter schools has demonstrated mixed results in their success. 2 

Although this legislation marks an important precedent in Tennessee policy, it will 
be critical in the coming years to determine and document the effectiveness of charter 
schools relative to traditional schools in promoting effective educational practices and 
raising student achievement. One of the most important foundational assumptions of 
charter schools is that the combination of autonomy and accountability will facilitate the 
development of innovative and high-quality instruction, which will then be implemented 
in the classrooms. These innovative, accountable, and autonomous schools will in turn, 
it is proposed, lead to improved student achievement as well as increased satisfaction 
among stakeholders (Bulkley & Fisler, 2002). However, in a review of the research on 
charter schools by RAND (Zimmer et al. , 2003), wide variation in student achievement 
results was found across schools. Accordingly, different groups have used the RAND 
report and other charter school research to support both positive and negative claims. 

The purpose of the present evaluation study was to examine the progress made 
in program implementation, school climate, and student achievement by the seven TN 
charter schools. A “mixed-methods” design, encompassing both qualitative and 
quantitative data, was employed, as will be described in the following sections. The 
questions upon which the evaluation methods are based relate to the progress of 
individual schools and the overall group in implementing desired strategies for 



2 The American Federation of Teachers (AFT) compiled data in August, 2004, ostensibly showing that 
students attending charter schools in the United States were not surpassing, and were even trailing, 
students attending regular schools on the NAEP tests (Page, 2004). These analyses, however, have 
been strongly criticized for restricted sampling and failing to adjust for student differences in ethnicity, 
prior achievement, and other characteristics. 
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curriculum, instruction, and organization, and in attaining the goals of No Child Left 
Behind by bringing every child to proficiency in reading and mathematics by 2014. 

In the following section, evaluation questions and the assumptions from which 
they were derived are presented. Table 1 shows each question with associated data 
collection tools, which in turn will be described in more detail in the Methodology 
section. One of the major evaluation areas dealing with the experiences of 
administrators and founders in establishing the charter school prior to school opening 
(see Table 1), has already been answered in a separate report (see Bates, Potter, Avis, 
McDonald, 2004). A summary of those findings will be provided below in establishing a 
framework for the present report. 

Charter School Assumptions and Associated Evaluation Questions 

1. Assumption : The major goal of school reform (i.e., specifically the creation of a 
charter school) is to raise student achievement and improve educational 
outcomes (attendance, socialization, etc.). 

■ Question: What are the immediate and long-term impacts of the charter 
school implementation on student achievement, attendance, dropout rates, 
behavior, and mobility? 3 

2. Assumption: High student achievement is fostered by effective teaching. 

■ Questions: (A) What is the frequency of usage of various traditional and 
alternative (student-centered) instructional strategies in the charter schools?; 
(B) What is the frequency of usage of selected instructional strategies 
associated with best practices?; (C) How do charter school climate outcomes 
compare to those reflected in national norms? 

3. Assumption: Positive educational outcomes at schools are fostered by positive 
school climate. 



3 At the time this report is being prepared (September, 2005), student-level data from the 2004-05 
Tennessee Comprehensive Assessment Program/Achievement Test (TCAP/AT) are not yet available. It 
is anticipated that such data will be released in October/November, 2005, at which time an addendum to 
this report will be prepared. 
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■ Questions: (A) What is the level of school climate at each charter school?; 
(B) Does school climate improve over time?; (C) How do charter school 
climate outcomes compare to those reflected in national norms? 

4. Assumption: To impact schools positively, educational reforms must be properly 
implemented. 

■ Question: To what degree and levels of quality are the identified goals and 
strategies of each charter school being implemented? 

5. Assumption: To be implemented effectively, educational reforms must have 
teacher support, sufficient professional development, and adequate resources. 

■ Questions: (A) What are teacher reactions to and experiences in TN charter 
schools?; (B) What are the adequacy and quality of professional development 
and needed resources? 

6. Assumption: The effectiveness and sustainability of educational reform and 
intervention programs are enhanced when students’ families are supportive and 
involved. 

■ Questions: (A) What are parent (caregiver) reactions to and experiences with 
TN charter schools?; (B) To what degree and in what ways are caregivers 
involved with each charter school? 

7. Assumption: Charter school personnel may face unique challenges in 

developing and establishing a functional charter school. 

■ Questions: (A) What are the experiences of the core-founding group in 
developing a charter school?; (B) What are the key strategies used during the 
pre-operation phase to successfully open a charter school? 

Summary of 2004 Start-Up Study 

Preliminary studies were conducted to examine the start-up (preoperational) 
processes of the four Tennessee charter schools that began operation in 2003 
(Robertson et al., 2003) and of the three additional schools that began operation in 
2004 (Bates et al., 2004). Interviews with each of the school’s principals and their 
sponsors (core founding groups) were conducted. Key findings that emphasize the 
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results from the most recent cohort are summarized below. It should be noted that 



start-up experiences were very similar across schools for both years of initiation. 

Several obstacles impacted one or more of the charter schools, including issues 
such as acquiring timely funding, acquiring building space, finalizing student enrollment, 
and hiring quality faculty. Two of these concerns reportedly stemmed from unclear 
charter school law and procedures. The first was that the timing of the state’s release of 
the high priority list conflicted with the starting date of school and subsequently hindered 
enrollment. The second was that the timing of the release of funds was not optimum to 
support initial startup costs. 

Several sources of support and continuing school needs were identified. The 
support for each of the schools came from local, state, and national sources. Common 
areas of support included the students and parents of the school, civic and community 
leaders, the school’s board of directors and sponsors, faith-based organizations, the 
Hyde Family Foundation, the Tennessee Charter School Resource Center, Memphis 
City Schools, and the TN Department of Education. School principals had differing 
responses to the ongoing needs of the school. The needs cited by the principals 
included time, student-teachers, technology, buildings, a library, corporate 
sponsorships, tutoring programs, faculty professional development, donation of goods 
and services, additional funding, and further parent education and involvement. 

The experiences of the second cohort of Tennessee’s charter school 
stakeholders resembled those of the first cohort. This is not surprising since the 
business of starting a school necessarily involves the same core components: 
establishing an organization, articulating a vision, securing funds, navigating 
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bureaucracies, identifying and preparing facilities, choosing leadership and staff, 
defining curriculum, implementing a community relations strategy, recruiting and 
admitting students, and obtaining the school building. While the experiences were new 
for the charters, they were not new for the district, state, and technical service 
supporters. According to the stakeholders who provided information for this report, 
there was evidence that both the state and the district provided focused technical 
support that alleviated some of the challenges facing the first cohort of charter schools. 
Nevertheless, systemic challenges remain. 

Summary of First Year Evaluation Findings 

In addition to the start-up studies, first year evaluations of the four schools that 
began operation in the 2003-04 academic year were conducted (Ross, McDonald, & 
Bol, 2004; Ross, McDonald, & Gallagher, 2005). The four schools were: Circles of 
Success Learning Academy (COSLA), Memphis Academy of Health Sciences (MAHS), 
Memphis Academy of Science and Engineering (MASE), and the Smithson-Craighead 
Academy (SCA). The same evaluation questions presented in the preceding section 
guided the evaluation of these schools after their initial year of operation. 

Achievement Outcomes. Scores on the Tennessee Comprehensive Assessment 
Program/Achievement Test (TCAP/AT) in Reading, Language Arts, and Math were 
used to measure achievement. The analyses of scores for COSLA, MAHS, and MASE 
showed directional advantages for the charter school students over their matched 
control counterparts on all three subtests. However, only the effect for MASE in Math 
was statistically significant. The median effect size was +0.15, indicating small to 
moderate effects. When the results for all three schools were combined, significant 
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advantages for the charter schools occurred on both Reading and Math, with a 
directional advantage in Language Arts. For SCA, the comparisons to control students 
yielded comparable to negative results. However, the SCA results need to be 
interpreted with caution because of the small sample of matched cases. Overall, these 
results were impressive given that the charter schools were only in their first year of 
operation and the transition of students to a new school is typically associated with 
lower achievement. 

Instructional Strategies. Classroom observation data revealed that all four 
schools primarily relied on teacher-centered instruction. With the exception of the 
percentages reported for MASE, the results obtained in this instructional category were 
typically higher than those observed in national samples. Moderate levels of student- 
centered instruction were observed across schools. Technology usage was not 
extensive at any of the four schools, and was considered weak at the two elementary 
schools (COSLA and SCA). The prevalence of higher-order strategies (e.g., higher 
level questioning and feedback) was markedly higher at SCA when compared to 
national norms and was found to be weak at MAHS. 

School Climate. Each of the four schools was characterized as having a strong 
school climate as measured by School Climate Inventory (SCI) scores. Nearly all of the 
mean ratings were higher than 4.0 (on the five point scale) for each of the dimensions 
comprising overall school climate. Two exceptions were somewhat lower mean ratings 
obtained for Order at MAHS and SCA. However, encouragingly, the mean ratings for all 
schools surpassed the national norms on the overall SCI score on each of the separate 
dimensions. 
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Implementation of Charter School Goals. Given that all schools were in their first 
year of operation, it was not surprising that most implementation benchmarks were 
judged to be in their initial phases. COSLA had the strongest evidence for more 
extensive implementation of curriculum and organization goals. In terms of 
implementing specific support goals, most of the schools were still in their early to 
intermediate stages. Lack of support was most apparent for criteria related to support 
from local and state education agencies as well as parent involvement. 

Teacher Perceptions and Resources. Teacher responses were mostly positive 
with respect to school mission or educational program and professional development. 
All schools were rated as moderate in the area of support. This intermediate rating was 
based on a perceived lack of support from state and local educational agencies, but a 
presence of support from other external agencies, the community, and in the case of 
COSLA, parental involvement. Although there were some mixed reactions to the 
availability and adequacy of resources, no school was rated as strong in the resource 
category because there was at least one area of resources judged as insufficient in 
each school. 

Parent Perceptions and Involvement. In general, parents appear to be very 
satisfied with the charter schools. Some common themes included the principals’ 
leadership as well as small school and class sizes that afford increased student safety, 
individual attention, and effective pedagogy. In three of the schools (COSLA, MAHS, 
and MASE) parents were dissatisfied with inadequate transportation services. There 
was some variation in the findings for parental involvement. Positive and consistent 
findings across sources were only obtained for COSLA. 
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Methodology 



Description of the Charter Schools 

As noted, four schools comprise the first cohort and three comprise the second 
cohort. Table 1 presents an overview of the schools by cohort. A more detailed, 
narrative description of each school follows the table. 

Table 1 

Overview of Schools 



School 


Cohort 


Level 


Grades 

(04-05) 


Enrollment 

(04-05) 


Location 


Circles of Success Learning Academy (COSLA) 


1 


Elementary 


K-3 


72 


Memphis 


Memphis Academy of Health Sciences (MAHS) 


1 


Middle 


6-7 


180 


Memphis 


Memphis Academy of Science & Engineering 
(MASE) 


1 


Secondary 


7-8 


107 


Memphis 


Smithson-Craighead Academy (SCA) 


1 


Elementary 


K-4 


148 


Nashville 


City University of School of Liberal Arts (CityU) 


2 


Secondary 


9 


121 


Memphis 


Star Academy 


2 


Elementary 


K-2 


116 


Memphis 


Yo! Academy 


2 


Secondary 


10-12 


131 


Memphis 



Circles of Success Learning Academy (COSLA). The Circles of Success 
Learning Academy (COSLA) is an urban charter school located in Memphis, 
Tennessee. During the 2004-2005 school year, COSLA served 72 students in grades K- 
3, with 91 .6% of the students eligible for free or reduced-price lunches. 

COSLA's staff consists of four teachers with a teacher-student ratio of 1:18. 
Other staff members include a principal, a curriculum facilitator, a Title I coordinator, an 
administrative assistant, and three teacher assistants. Thirty percent of the staff 
responding to the School Climate Inventory described their educational attainment as 
culminating in an associates degree, 50% as culminating in a bachelor’s degree, 10% 
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as culminating in a Master’s degree, and 10% as culminating in a doctorate degree. The 
majority of the staff (60%) have worked at the school for the two years of its operation, 
and 40% have worked at the school for one year. 

The general focus at COSLA is literacy development. With its school-wide 
program, attempts are made to integrate literacy across subject areas. The school 
utilizes scientifically-based methods and strategies in the delivery of the curriculum. 
Accordingly, curriculum content and performance standards are aligned with current 
school district and state assessment goals. COSLA also seeks to ensure that teachers 
use a variety of teaching strategies in the delivery of instruction. Organizational 
structures are designed to facilitate high levels of learning and therefore include the 
systematic monitoring of grade level planning, classroom teaching practices, and 
student progress. Support structures include feedback from the principal and the 
curriculum facilitator, the modeling of effective practices, regular professional 
development opportunities for all staff, and both internal and external support services. 

Memphis Academy of Health Sciences (MAHS). The Memphis Academy of 
Health Science (MAHS) is located in an urban area of Memphis, Tennessee. The 
school is housed in the upper floor of Caldwell Elementary School. There are 180 
students attending grades six and seven. African-American students make up 100% of 
the school’s population. According to school administrators, 75% of the students receive 
free or reduced-price lunches. 

The school’s faculty and staff consist of 12 teachers (including one Special 
Education teacher), one principal, and one secretary. The administration reports the 
teacher-student ratio as 1 :20. Based on responses to the School Climate Inventory, 
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approximately one-third of the respondents (36.4%) have earned masters degrees and 
the remaining staff members (63.6%) have bachelor degrees. Most (72.7%) of the 
faculty and staff have less than six years of experience working in school settings. 
Another 18.2% have six to ten years experience, 9.1% have 11-15 years experience, 
and none has more than 15 years experience. Approximately one-third (36.4%) of the 
faculty and staff have had two years experience working in this school, and a majority 
(63.6%) have only one year experience given that a new grade (7) was added this year. 

The curriculum is a standards-based program that incorporates interdisciplinary 
projects and experiential learning centered on a health science theme. Benchmarks 
established by the school for its second year of operation primarily focus on establishing 
structures that enable effective implementation and evaluation of the instructional 
program, effective communications between stakeholders, and a positive environment 
for teaching and learning. 

Memphis Academy of Science and Engineering (MASE). The Memphis 
Academy of Science and Engineering (MASE) is housed in an office building in 
downtown Memphis. The school is on the two upper floors of the building, with the 
seventh grade inhabiting one floor and the eighth grade the floor above. 

During the 2004-2005 school year, there were 107 seventh-grade students and 
144 eighth-grade students. The students were predominantly African-American (97%) 
with a few White (2.3%) and Hispanic (0.7%) students. Free or reduced-priced lunches 
were provided for 68% of the students. A total of 24 students left MASE during the 
school year, of which 12 left within the first two weeks because they had been accepted 
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by other schools. The others left of their own accord, moved out of the district, or were 
asked to leave. 

There were 12 teachers at MASE this year. There were also two support staff 
and three administrators, and one person with both administrative and support duties. 
The administration consisted of a principal, a vice principal, an educational specialist, 
and a technology specialist/support person. 

The student-teacher ratio was 21:1. Eight of the teachers had master’s degrees 
and the other four had bachelor’s degrees. The staff members had bachelor’s degrees 
and the administrators all had master’s degrees. The teachers had experience levels 
from first year (one teacher), fourth year (two teachers), sixth year (four teachers), 
seventh year (four teachers), and 22 nd year (one teacher). Ten teachers, one 
administrator, and one support staff member were new to MASE. Two teachers, two 
administrators, one support person, and the technology specialist/support person were 
in their second year. 

The program includes longer days, Saturday school, high technology use, 
continuous monitoring of student progress, and an intense focus on core curricula 
integrated across subjects. Mathematics, science, and language arts are emphasized 
using authentic, varied, and integrated instruction and assessment strategies. The 
schedule entails a full day of school (8 - 5pm) so that all classes could meet twice. The 
students extended, practiced, finished, or redid the morning’s lessons in the afternoon. 
Community projects (excursions) were added to allow for real-life applications of the 
students’ work. 
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Smithson-Craighead Academy. Smithson-Craighead Academy, an urban school 
in Nashville, TN, serves 148 students in kindergarten through fourth grade. The 
students are predominantly (99%) African-American. All students are eligible for free 
(90%) or reduced-price (10%) lunches. The student body is classified by the 
administrators and teachers as “at-risk.” 

Staff and faculty of the school include ten teachers and five teaching assistants. 
Other staff members include the principal, a curriculum coordinator, and four specialists: 
librarian, special education teacher, computer instructor, and computer/special 
education coordinator. The founder, Sister Sandra Smithson, provides administrative 
and instructional consultation. This spring (2005), an additional staff member was hired 
to serve as a liaison between parents and school personnel. The majority (73.7%) of 
faculty/staff report that their education culminated in either a bachelor’s (31 .6%) or a 
master’s (42.1%) degree. Experience as a school employee ranges from five years or 
less (57.9%) to over 20 years (21.1%). Over half (68.4%) have 1-10 years of 
experience. The same percentage (68.4%) have more than one year of experience at 
SCA. 

The focus of the school is to meet the academic and social needs of at-risk 
children. Students are taught three goals: self-control, work before play, and obey your 
teachers. They recite these goals on a daily basis. The curriculum for the lower grades 
includes benchmark testing to ensure mastery of basic skills before proceeding to more 
advanced skills. The scope and sequence of the curriculum is based on both research- 
supported practices and state curriculum standards. Teachers are given the latitude to 
use innovative strategies to meet the school’s educational goals. A daily, after-school 



22 




program is available to provide students with additional educational support. Many 
teachers and administrators participate in the after-school program. 

City University School of Liberal Arts. City University School of Liberal Arts 
(CityU) is located in an urban area of Memphis, Tennessee. The school is housed in 
Greater Middle Baptist Church. In this first year of operation, there are 121 students 
attending grade 9, the only grade served presently. African-American students 
constitute 98.3% of the school’s population. There was also one Caucasian and one 
Hispanic student. According to the school’s administration, 78% of students are eligible 
for free or reduced-price lunches. 

The school’s faculty and staff consist of eight teachers, one principal, one office 
manager, one instructional leader, and one part-time counselor. The administration 
reports that the teacher to student ratio for core classes is 1 :20 and for non-core classes 
is 1:30. Of those responding to the School Climate Inventory, 62.5% of the faculty and 
staff described their educational attainment as culminating in bachelor’s degrees, 12.5% 
as culminating in master’s degrees, and 25% as including a degree beyond Master’s. 
One-fourth (25%) of the faculty and staff have more than 20 years of experience as 
school employees, while one-half (50%) have less than 1 1 years of experience. 
Because CityU is a new charter school, all faculty and staff have one year of experience 
working specifically in this school. 

The curriculum developed and employed at CityU has a liberal arts core and is 
aligned with College Board standards to build a strong college-oriented focus. This 
theme is continued throughout CityU’s operations, with teachers being referred to as 
“professors” and students as “scholars.” Students change classes after each lecture, 
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not when a bell rings, and school policies reflect a more collegiate atmosphere (such as 
allowing cell-phone usage between classes). The educators at Cityll want their students 
to be prepared for college and are attempting to provide them with an opportunity to 
learn in an atmosphere they would not experience in most high schools. 

Star Academy. Star Academy is located in the northwest region of Shelby 
County in the urban outskirts of Memphis, Tennessee. Star Academy serves 116 
students in kindergarten through second grade. The student population is predominantly 
(98%) African-American, with approximately 90% of the students qualifying for free or 
reduced-priced lunch. 

There are six classroom teachers at Star Academy, plus a Title I Coordinator and 
Curriculum Coordinator. There is a 20:1 student-teacher ratio across classrooms. All 
professional faculty maintain licenses in their assigned area. One-third of the 
respondents to the School Climate Inventory indicated that they have a bachelor’s 
degree while the remaining two-thirds indicated having at least a master’s degree. 
Slightly less than half of the respondents (44%) indicated that they have worked in a 
school-based setting for six to ten years, while one-third indicated that they have more 
than 20 years of experience. Only 11% reported having less than six years of 
experience. 

In its first year of operation, Star Academy’s primary area of focus was on 
Reading and Language Arts. The McGraw-Hill series provides the core curricular 
materials in reading and math. For students identified as needing extra support, 
supplementary programs are used (Head Sprout and Failure Free Reading). A variety of 



24 




instructional strategies, such as cooperative learning, peer tutoring, and direct 
instruction, are employed. 

Yo! Academy. Yo! Academy of Visual and Performing Arts is located in 
southwest Memphis in an industrial warehouse area. In its first year, the school initially 
held classes off-site. Academic classes met in space leased from a church 
approximately six miles north of the school building, while visual and performing arts 
classes met in the facility that housed the city’s Youth Opportunity Movement program. 
By midyear, the school’s own facility was adapted for secondary school use so that all 
classes could be conducted under a single roof. The school operates with a year round 
calendar that includes one Saturday per month. 

During the first year of operation, the school served only students in tenth, 
eleventh, and twelfth grades. The school is scheduled to add ninth grade in the second 
year of operation. 

The school opened in July 2004, with 131 students. All students were African- 
American and were eligible for free or reduced-price meals. Nearly half of the students 
(49%) were tenth graders, over a third (37%) were eleventh graders, and the remainder 
(14%) were twelfth graders. By the end of the year, the tenth-grade class had grown by 
seven students, while the eleventh-grade class lost eight and the twelfth-grade class 
lost six students. 

The faculty consists of one principal, one assistant principal, seven teachers, two 
office staff members, and a case manager/psychologist. All are African-American. Both 
the principal and assistant principal also teach classes. All of the teachers have 
teaching certification and the current principal is in the final stages of a doctoral degree 
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in school leadership. Two teachers teach one course for which they are uncertified. 
Since the beginning of the school year, the school has had four different principals. In 
addition, the original executive director of the sponsoring agency (Yo! Memphis 
Foundation) has been replaced. 

Of the faculty and staff members responding to the School Climate Inventory, 
four reported having master’s degrees and two reported having a degree beyond a 
master’s. Four reported having between six and ten years of experience working in a 
school setting, while the remaining three had less than six years of experience. One 
teacher had taught in a charter school before Yo! Academy. 

The mission of the school is to provide a rigorous academic preparation for 
students while incorporating the visual and performing arts, all within a safe 
environment for youths who are considered to be at-risk. Yo! Academy was established 
on the foundation of work done in Memphis under a five year demonstration program 
funded by the U. S. Department of Labor called the Youth Opportunity Movement. 
Funding for that program concluded June 30, 2005. The school’s focus on visual and 
performing arts is a direct outgrowth of the success of the Youth Opportunity Memphis 
(“Yo! Memphis”) program. In the Academy’s first year, over half of its students had 
experience with Yo! Memphis. The director of the Yo! Memphis Show Choir continues in 
that capacity at the Academy while serving as the school’s assistant principal. 
Instrumentation 

Instruments used for data collection were developed and validated by the Center 
for Research in Educational Policy (CREP) as part of the Formative Evaluation for 
School Improvement Process (FEPSI) (see www.memphis.edu/crep ). A description of 



26 




each instrument is provided below. A summary of the instrumentation employed to 
address each research question is provided in Table 2. The standard data collection 
schedule is shown in Table 3. 

Table 2 

Evaluation Questions by Instrument 



Evaluation Questions 


Instruments 


1 . What are the immediate and long-term impacts of the 
charter school implementation on student 
achievement, attendance, drop-out rates, behavior, 
and mobility? 


Analysis of achievement data and archival data 
(attendance, discipline referrals, transfers, drop- 
out rates) 



2. What is the frequency of usage of various traditional SOM 
and alternative instructional strategies? RSCA 



What is the quality of usage of selected instructional 
strategies associated with best practices? 



How do charter school outcomes compare to those 
reflected in national norms? 

3. What is the school climate at the charter school? 

Does school climate improve over time? 

How do charter school climate outcomes compare to 
those reflected in national norms? 

4. To what degree and levels of quality are the goals and 
strategies of the charter school being implemented? 



5. What are teacher reactions to and experiences in the 
charter school? 

What are the adequacy and quality of professional 
development and resources? 

6. What are parent (caregiver) reactions to and 
experiences with the charter school? 

To what degree and in what ways are caregivers 
involved with the school? 

7. What are the experiences of the core-founding group 
in developing a charter school? 



School Climate Inventory (SCI) 
Teacher Focus Group 
Student Focus Group 
Principal Interview 



Implementation Benchmarks 

Teacher Focus Group 

Principal Interview 

CSTQ 

SOM 

RSCA 

CSTQ 

Teacher Focus Group 
Principal Interview 



Parent Survey 
Principal Interview 
Teacher Focus Group 
Student Focus Group 

Core Founding Group (School Design Team) 
Survey/Interview 



What are key strategies used during the pre-operation 
phase to successfully open a charter school? 
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Table 3 

Data Collection Outline 



Instrument 


Timeline 


Description 


Core Founding Group Survey/Interview 


Summer 2004 


Survey and interview of key planning/pre- 
operational phase personnel 


Benchmark training, development, 
revision 


Fall 2004 


Train school Principal and Staff; develop draft 
document 


SOM 


Fall 2004; 
Spring 2005 


6 three-hour observations 


Rubric for Student-Centered Activity 
(RSCA) 


Fall 2004; 
Spring 2005 


Same as SOM Data Collection 


Principal Interview 


February 2005 


1 hour interview 


Charter School Teacher Questionnaire 
(CSTQ) 


March 2005 
(Faculty Meeting) 


Teachers complete the questionnaire (part- 
time faculty can also complete) 


School Climate Inventory (SCI) 


March 2005 
(Faculty Meeting) 


Professional staff complete inventory (part- 
time faculty/staff can also complete) 


Parent Survey 


Spring 2005 


Parents/caregivers complete the survey 


Teacher Focus Group 


Spring 2005 


1 hour group interview 


Student Focus Group 


Spring 2005 


30-50 minute group interview 


Benchmarking Review 


May 2005 


Principal/Staff review progress 



School Climate Inventory (SCI). The School Climate Inventory (SCI) was 
developed by researchers at the Center for Research in Educational Policy, at The 
University of Memphis in 1989. Since that time, the instrument has been used for 
school-based improvement planning in schools and school districts in several states, 
and has proven to be especially valuable for the monitoring and management of change 
initiatives. 

The SCI consists of seven dimensions logically and empirically linked with factors 
associated with effective school organizational climates. Each scale contains seven 
items, with 49 statements comprising the inventory. Responses are scored through the 
use of Likert-type ratings [strong disagreement (1) to strong agreement (5)]. Each scale 
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yields a mean ranging from 1 to 5 with higher scores being more positive. Additional 
items solicit demographic information. 

Face validity of the school climate items and logical ordering of the items by 
scales were established by the research team during the development of the inventory 
(Butler and Alberg, 1991). Subsequent analysis of responses collected through 
administration of the inventory in a variety of school sites substantiates validity of the 
items. Scale descriptions and current internal reliability coefficients, obtained using 
Cronbach's alpha, appear in Table 4. 



Table 4 

School Climate Inventory Internal Reliability and Scale Descriptions 



Scale 


Internal 

Reliability 


Description 


Order 


a=.8394 


The extent to which the environment is ordered and 
appropriate student behaviors are present 


Leadership 


a=.8345 


The extent to which the administration provides 
instructional leadership 


Environment 


a=.8094 


The extent to which positive learning environments 
exist 


Involvement 


a=.7582 


The extent to which parents and the community are 
involved in the school 


Instruction 


a=.7453 


The extent to which the instructional program is well 
developed and implemented 


Expectations 


a=.7275 


The extent to which students are expected to learn 
and be responsible 


Collaboration 


a=.741 7 


The extent to which the administration, faculty, and 
students cooperate and participate in problem 
solving 



School Observation Measure (SOMP). The SOM was developed to determine 
the extent to which different common and alternative teaching practices are used 
throughout an entire school (Ross, Smith, & Alberg, 1998). The standard, or whole- 
school, SOM procedure involves observers’ visiting 10-12 randomly selected 
classrooms, for 15 minutes each, during a three-hour visitation period. The observer 
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examines classroom events and activities descriptively, not judgmentally. Notes are 
taken relative to the use or nonuse of 24 target strategies. At the conclusion of the 
three-hour visit, the observer summarizes the frequency with which each of the 
strategies was observed across all classes on a data summary form. The frequency is 
recorded via a 5-point rubric that ranges from (0) Not Observed to (4) Extensively. Two 
global items are used to rate, respectively, the level of academically-focused 
instructional time and the degree of student attention and interest. 

The SOM strategies include traditional practices (e.g., direct instruction and 
independent seatwork) and alternative, predominately student-centered methods 
associated with educational reforms (e.g., cooperative learning, project-based learning, 
inquiry, discussion, technology use as a learning tool). The strategies were identified 
through surveys and discussions involving policy makers, researchers, administrators, 
and teachers, as those most useful in providing indicators of schools’ instructional 
philosophies and implementations of commonly used reform designs (Ross, Smith, 
Alberg, & Lowther, 2004). 

To ensure the reliability of data, observers receive training, a manual providing 
definitions of terms, examples and explanations of the strategies, and a description of 
procedures for completing the instrument. After receiving the manual and instruction in 
a group session, each observer participates in sufficient practice exercises to ensure 
that his/her data are comparable with those of experienced observers. In a reliability 
study (Lewis, Ross, & Alberg, 1999), pairs of trained observers selected the identical 
overall response on the five-category rubric on 67% of the items and were within one 
category on 95% of the items. Further results establishing the reliability and validity of 
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SOM are provided in the Lewis et al. (1999) report. In a reliability study using 
Generalizability Theory, Sterbinsky (2003) found reliability at the .74 level for 5 SOMs 
conducted at a school. Reliability increased to .82 with 8 SOMs and to .85 with 10 
SOMs conducted at a school. 

Rubric for Student-Centered Activities (RSCA). The Rubric for Student-Centered 
Activities was developed by CREP (Lowther, Ross, & Plants, 2000) as an extension to 
SOM. The RSCA is used by observers to more closely evaluate the degree of learner 
engagement in seven selected areas considered fundamental to the goals of increasing 
student-centered learning activities (cooperative learning, project-based learning, 
higher-level questioning, experiential/hands-on learning, student independent 
inquiry/research, student discussion, and students as producers of knowledge using 
technology). These strategies reflect emphasis on higher-order learning and attainment 
of deep understanding of content, and whether or not technology was utilized as a 
component of the strategy. Such learning outcomes seem consistent with those likely 
to be engendered by well-designed, real-world linked exercises, projects, or problems 
utilizing technology as a learning tool. 

Each item on the RSCA includes a two-part rating scale. The first is a four-point 
scale, with 1 indicating a very low level of application, and 5 representing a high level of 
application. The second is a Yes/No option to the question: “Was technology used?” 
with space provided to write a brief description of the technology use. The RSCA was 
completed as part of SOM observation periods. 

To ensure the reliability of data, observers receive training consisting of terms, 
examples, and explanations of the target strategies, and a description of procedures for 
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completing the instrument. After receiving the manual and instruction in a group 
session, each observer participates in sufficient practice exercises to ensure that his/her 
data are comparable with those of experienced observers. 

Charter School Teacher Questionnaire. This questionnaire was designed to 
assess teacher perceptions about the school in the areas of professional development, 
support, pedagogical change, and outcomes. Included on the questionnaire are 20 
closed-ended items using a five-point Likert-type scale (from strongly disagree to 
strongly agree), and the following four open-ended questions: What do you see as 
positive or most successful aspects of your charter school?; What do you consider to be 
negative aspects or areas in need of improvement at your school?; In your opinion, 
what makes a charter school (like this one) different from a regular public school?; and, 
Any other comments you would like to make regarding your experiences as a charter 
school teacher? 

Charter School Parent Questionnaire. This instrument was designed to obtain 
parent perceptions of the school in areas such as instruction, curriculum, 
communication, and opportunities for involvement. Included on the questionnaire are 
17 closed-ended items using a five-point Likert-type scale (from strongly disagree to 
strongly agree), and the following four open-ended questions: What are the strengths of 
this school?; What would you like to see improved at this school?; In your opinion, what 
makes a charter school (like this one) different from a regular public school?; and, Any 
other comments you would like to make about this school or charter schools in general? 

Focus groups and interviews. To supplement the survey data, site researchers 
at each charter school conducted a principal interview, a teacher focus group, and a 
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student focus group. The time period for each was approximately 30 - 60 minutes for 
each interview. Teacher and student participants were randomly selected to participate. 
A semi-structured protocol, involving standard questions with flexibility for follow-up on 
selected responses, was used. In all three protocols the basic questions concerned 
experiences during the year, differences from regular (non-charter) schools, reactions to 
major school components (e.g., teaching methods, curriculum, parent involvement), 
perceived strengths and weaknesses, and recommendations for improvement. 
Procedure 

A “site researcher” from CREP was assigned data collection responsibility for 
each charter school. This individual and support research staff visited the assigned 
school several times during the year to conduct the SOM visits, administer the 
questionnaires, and conduct the interview and focus groups. In addition, the site 
researcher worked with the school leadership team to develop “implementation 
benchmarks” describing beginning, intermediate, and full implementation phases and 
associated evidence indicators for major school components in the areas of curriculum, 
instruction, and organization. At the end of the year, the site researcher met again with 
the leadership team to determine and identify the phase that had been achieved for 
each benchmark. Data from all instruments were then used to prepare individual school 
formative evaluation reports indicating status and progress during the year, as well as 
the present evaluation report. 
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Results 



In this section, the second-year findings from the first cohort of schools and the 
first year findings from the second cohort are summarized separately and then 
integrated to address each research question. There are two exceptions: Research 
Question 1 (see Table 2), regarding student achievement will be addressed in a 
separate report prepared in late fall, 2005. Research Question 7, regarding the 
experiences of leaders and founders in opening the charter school, has already been 
addressed in detail in separate reports (Bates et al., 2004; Robertson et al., 2003) and 
also summarized in the introduction to the present report. 

Question 2. What is the frequency of usage of various traditional and alternative 
(student-centered) instructional strategies in the charter schools and compared to 
national norms? 

Observational data collected via the School Observation Measure (SOM) and the 
Rubric for Student-Centered Activities (RSCA) was used to gauge the frequency with 
which various traditional and alternative instructional strategies were observed in the 
charter schools. More specifically, patterns in results were organized under four 
different categories of instructional orientation and strategies: teacher-centered, 
student-centered, technology usage, and higher-order instruction. The findings from the 
observations conducted in the charter schools were compared to the national normative 
data available for the SOM. For reference purposes, these norms are provided in 
Appendix A. Because the RSCA was more recently developed, normative data are not 
yet available for this instrument but will be used to supplement the SOM data. In 
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addition to comparing the observational data to norms, we also noted trends in the 
results across years for those charter schools in their second year of operation. 

Circles of Success Learning Academy. As we reported during COSLA’s first year 
of operation, direct instruction was the most frequently observed instructional strategy. 
In fact, the percentage of time direct instruction was frequently to extensively observed 
increased from 67% in the first year to 83% in the second year. These percentages 
were higher than the national elementary school norms. In contrast to the increase in 
direct instruction, the percentage of time students were engaged in independent 
seatwork, another more traditional strategy, was low relative to national norms and to 
the previous year. With regard to student-centered strategies, the percentage of time 
allotted to cooperative/collaborative learning remained constant across years and was 
higher than in the normative sample. Additionally, increases were observed in teachers’ 
adoption of the role of coach or facilitator, and the findings compared favorably with 
national norms. Based on data obtained from the RSCA, student-centered activities, 
especially cooperative learning and higher level questioning, were implemented with 
greater quality than in the first year. However, findings suggest no use or very limited 
use of technology. 

Memphis Academy of Health Sciences. Much of the instruction observed at 
MAHS was traditional and teacher-centered. The percentage of time devoted to direct 
instruction remained high in the second year and was somewhat higher than the 
normative data for secondary schools. The strategy was used at least occasionally 
100% of the time in MAHA classrooms compared to 90% in the national sample. 
Student engagement in independent seatwork was frequently observed but these 
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figures were more comparable to the normative data. Although student-centered 
strategies were used more frequently than in the first year, their application was still 
limited. For example, cooperative learning was observed more frequently in the second 
year but the frequency or levels of use were low in an absolute sense and when 
compared to normative data. This pattern of results was also found for higher-order 
instructional strategies and use of technology. That is, there was some increase from 
the first year, but the results still reflected low levels of implementation that did not 
compare favorably with national norms. A notable exception to this trend was computer 
use for instructional delivery. This use of technology was occasionally used 50% of the 
time, and this value was higher than the percentage observed in other secondary 
schools. 

Memphis Academy of Science and Engineering. Although there was a decrease 
in teacher-centered strategies when compared to last year, time spent in direct 
instruction remained high. The results were similar to levels observed in secondary 
schools nationally. A similar declining trend was noted for independent seatwork. The 
strategy tended to be frequently rather than extensively observed when contrasted with 
the previous year. Higher-order strategies were more frequently observed. Specifically, 
we observed more high-level instructional feedback and questioning when compared to 
the previous year and to national norms. Progress in increasing student-centered 
instruction was evidenced by observed increases in teacher acting as coach or 
facilitator, project based learning, experiential learning, and independent inquiry or 
research. However, it should be noted that these trends in student-centered strategies 
mostly represented a move from not observed to rarely or occasionally observed, and 
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they still occurred at lower rates than for national samples. In contrast, technology use 
not only increased this year, but the percentages exceeded those reported in the 
normative sample. This was especially true for its implementation as a learning tool or 
resource. While there was an increase in the frequency of technology usage, RSCA 
data indicates that its application tended to be limited when used in conjunction with 
student-centered activities. 

Smithson-Craighead Academy. With regard to teacher-centered instruction, we 
observed a decrease in the time spent in direct instruction. Direct instruction was still 
employed at occasional to frequent levels, but these levels were more comparable than 
in Year 1 to national norms. Independent seatwork was also frequently observed but 
less extensively than in the previous year and at rates comparable to those in the 
normative sample. There was frequent use of higher-order strategies when compared 
to the national sample, yet there was an overall decrease in their usage levels in 
comparison with last year. For instance, higher-level instructional feedback was used at 
least occasionally during all of the visits last year, but this percentage dropped to 50% 
of the time for the second year. RSCA data shows a similar decline in student-centered 
activities in terms of both frequency and quality of application. More encouragingly, 
there was an increase in the use of technology compared to the previous year. SOM 
data suggests more frequent use of computers for instructional delivery and technology 
for learning or resources, with rates higher than those reported in the national norms for 
elementary schools. 

City University School of Liberal Arts. The most frequently observed instructional 
strategy was direct instruction. It was frequently to extensively observed about 83% of 
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the time, and this percentage is somewhat higher than the national norm for secondary 
schools. The second most frequent instructional strategy was independent seatwork, 
and these rates were comparable to national norms. Student-centered strategies were 
observed less frequently than teacher-directed strategies with most student-centered 
strategies not observed or rarely observed. Technology use was not observed in any of 
the classrooms during the observation visits. Compared to national norms, City U had 
much lower rates of student-centered instruction or technology use in their classrooms. 
However, some higher-order instruction occurred at rates similar to those observed 
nationally. Higher-level feedback and questioning were both observed at least 
occasionally about 50% of the time. 

Star Academy. Although direct instruction was the dominant instructional 
strategy observed, the student-centered strategies of cooperative/collaborative learning 
and teacher as coach or facilitator were also commonly employed. The percentage of 
time spent in these three categories of instruction was higher than the national norms 
obtained for elementary schools. Independent seatwork was another prevalent strategy 
used and was observed at least occasionally during all of the visits. The rates for this 
strategy were also higher than those reported in national norms. Engagement in higher- 
order strategies was evident from both the SOM and RSCA data. The use of higher- 
level instructional feedback was observed occasionally or frequently during half of the 
visits, a rate modestly higher than in the normative sample. Technology use was not 
observed or at best, rarely observed. 

Yo! Academy. Traditional instructional strategies were primarily observed at Yo! 
Academy. The most dominant strategies were direct instruction and independent 
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seatwork, and they occurred more frequently than in the national sample of secondary 
school classrooms. Few student-centered instructional strategies were employed with 
at least occasional frequency. One exception was the teacher acting as coach or 
facilitator, yet this rate occurred at a lower rate than in the normative sample. There 
was some indication of the use of higher-order pedagogical strategies. Higher-level 
questioning strategies were occasionally used about 33% of the time. Results for this 
strategy compared favorably to the national sample. However, the frequencies for 
technology use did not compare favorably with normative data. Technology as a 
learning tool or resource was rarely observed and computer use for instructional 
delivery was seen no more than occasionally during 20% of the visits. 

Summary findings across schools. Table 5 presents the summary findings by 
school in four major categories associated with instructional orientation and strategies. 
All seven schools were rated as strong with respect to teacher-centered instruction. The 
rating was primarily due to the findings that direct instruction was the predominant 
instructional orientation. These rates were typically higher or comparable to rates 
observed in the national samples. It should be noted that the percentage of time spent 
in direct instruction is high in both the elementary and secondary school classrooms 
nationally. Four of the schools (COSLA, MASE, SCA, and Star) employed student- 
centered instruction to a moderate degree. The most frequent types of student- 
centered instruction included cooperative/coliaborative learning and teacher acting as 
coach facilitator. This trend makes logical sense because teachers are more inclined to 
assume a coaching role as students work together in groups. A wide array of student- 
centered strategies was observed at these schools, albeit at a lower frequency than the 
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previously mentioned strategies. Three of the charter schools were categorized as 
weak in this category (MAHS, CityU, and Yo!) due to low levels of occurrence and 
engagement in an absolute sense and relative to national norms. All but one school 
(MAHS) implemented higher-order strategies at moderate levels. The frequencies were 
not consistently high, but they compared favorably to the normative samples. MAHS 
was considered weak because high-level feedback and questioning were not commonly 
observed, and their rates of occurrence were much lower than those reported nationally. 
The three schools in their first year of operation (CityU, Star, and Yo!) were rated as 
weak in technology use. Technology use for any purpose never or rarely occurred in 
these schools. Technology use also remained weak in COSLA where a similar pattern 
was observed. In contrast, technology use increased in three of the first cohort of 
charter schools (MAHS, MASE, SCA). Although the percentages compared favorably 
with normative samples, RSCA data indicated that the level of application was not yet 
strong in intensity or quality. 
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Table 5 

Summary Ratings for Schools on Teaching Orientations 



Indicator 



School 



COSLA 

MAHS 

MASE 

SCA 

CityU* 

Star* 

Yo!* 



Teacher- 

centered 



*First year schools 



□ 

Beginning/Low Usage 



Student- 

centered 



Higher-order 

strategies 



Technology 
usage 




Moderate 




Strong 



Question 3. What is the school climate at the charter schools and how does the 
climate compare to national norms? 

To address this research question, we primarily relied on the rating scale data 
obtained from the School Climate Inventory (SCI). Because we compared the SCI 
results for the charter schools with national norms, the national averages for elementary 
and secondary schools (middle and high schools) are presented in Table 6. The 
interpretation of SCI data was augmented by the qualitative responses obtained from 
focus group interviews, principal interviews, and open-ended items on the SCI. Any 
themes related to the dimensions of school climate were identified and used to 
illuminate the quantitative findings. 
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Table 6 

School Climate Inventory (SCI-R) Dimension Averages for Elementary and Secondary Schools 



Dimension 


COSLA 




SCA 




Star 


Elementary 

Norm 


Collaboration 


4.63 




3.79 




4.62 


3.88 


Environment 


4.37 




3.84 




4.73 


3.96 


Expectations 


4.71 




4.25 




4.78 


4.04 


Instruction 


4.66 




4.21 




4.70 


4.17 


Involvement 


4.64 




3.95 




4.50 


3.91 


Leadership 


4.75 




4.15 




4.90 


4.08 


Order 


4.39 




3.23 




4.59 


3.50 


OVERALL 


4.58 




3.94 




4.69 


3.93 




Dimension 


CityU 


MAHS 




MASE 


Yo! 


Secondary 

Norm 


Collaboration 


3.89 


4.09 




4.20 


3.96 


3.71 


Environment 


3.88 


3.77 




4.06 


3.80 


3.73 


Expectations 


4.20 


4.29 




4.57 


4.12 


3.82 


Instruction 


4.29 


4.34 




4.43 


4.20 


4.06 


Involvement 


4.29 


3.86 




4.06 


3.60 


3.63 


Leadership 


4.20 


4.08 




4.24 


3.98 


3.94 


Order 


3.93 


3.47 




3.68 


3.60 


3.26 


OVERALL 


4.10 


4.00 




4.15 


3.93 


3.73 



Circles of Success Learning Academy. SCI findings suggest that school climate 
continues to be very positive at COSLA. The overall mean rating was 4.58 with means 
for each dimension ranging from a low of 4.37 for Environment to a high of 4.75 for 
Leadership. Compared to the initial year, the mean ratings increased on six of the 
seven dimensions of climate. Although the mean rating for Environment was lowest 
(4.37), this area did increase from 4.18 last year. A small decrease was observed for 
Involvement from 4.74 to 4.64, though this area remains positive overall. 

The mean ratings also continue to compare favorably with national norms. The 
SCI mean ratings for COSLA exceeded the means obtained for the normative sample of 
elementary schools on each dimension. In most cases these differences were 
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substantial. For example, COSLA’s mean rating for 4.63 for Collaboration can be 
contrasted with the national average of 3.88. 

The positive SCI findings were confirmed by qualitative interview responses. 
Student responses provided insight into the school climate at COSLA and perhaps, the 
increased mean ratings on the Order dimension. Their responses reflect a great deal of 
respect and caring among teachers and students that appears to impact classroom 
management. Some described their prior schools as very punitive and appreciated the 
more positive approach to discipline in their new school. “I used to get paddled but 
teachers here give us a second chance or give us the bad grade we deserve.” Another 
student said, “Teachers at COSLA don’t accuse us. They say, ‘if you fall short, you can 
lift back up.” Similarly, teachers characterized the relationships among faculty and 
teachers as familial. They were also laudatory about school leadership, collaboration, 
and the creativity associated with a flexible teaching and learning environment. 

Memphis Academy of Health Sciences. School climate scores decreased from 
Year 1 to Year 2 at MAHS, yet the mean ratings were still largely positive as evidenced 
by the overall mean of 4.00 (compared to national norm of 3.73). The highest mean 
rating was obtained for the Instruction dimension, and the lowest was for the Order 
dimension. An examination of individual items revealed that student tardiness or 
absence contributed to the low score on this dimension. Notably, only 9% of 
respondents agreed that tardiness and absence was not a problem this year as 
compared to nearly 80% agreement last year. Another substantial decrease in scores 
was found for the Involvement dimension and this decrease was partly due to a 
perceived lack of perceived support from parents (from 100 to 27% agreement). 
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Although decreases in ratings were observed across years, the means still 
compared favorably with national averages reported for secondary schools. MAHS 
mean ratings were higher on each dimension and overall. In some cases, these 
differences were large (e.g., 4.29 versus 3.82 for Expectations) but in others more 
modest (3.77 versus 3.73 for Environment). 

The available qualitative data point to a positive school climate. The principal 
strives to “create ownership” of the school by staff and students, and this theme also 
emerged in the teacher focus group interview. They described creating goals and 
expectations for MAHS with a commitment to student success and school improvement 
over time. They described the school environment as having a “family atmosphere” and 
used terms such as “intimate”, “comfortable”, “welcoming”, and “warm” to characterize 
their school. Students noted the high expectations set by their teachers and principal. 
“The teachers and principal are strict. They push us to do what we’re supposed to do.” 
The qualitative responses did not yield clues as to why SCI ratings decreased across 
years. The deceases may have been influenced by the addition of another grade level 
this year and having nearly twice as many students enrolled. 

Memphis Academy of Science and Engineering. The mean SCI ratings 
suggested a positive school climate at MASE. The overall mean rating was a 4.15 on 
the 5- point scale compared to the national overall norm of 3.72. In fact, only Order fell 
below a 4.00 mean rating. The highest mean was obtained on the Expectations 
dimension. While still positive, the means on each dimension were lower than those 
obtained in the previous year. This decline was particularly apparent for the Order 
dimension (from 4.70 to 3.68). The expansion of the school in terms of student 
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enrollment (from 147 to 252) and grade levels served (from 7 th to 7 th and 8 th ) may have 
influenced these results. Percentages of agreement obtained on individual items 
comprising the Order scale suggest that student discipline problems may adversely 
affect the school climate at MASE. Notably smaller percentages of respondents agreed 
that rules for student behavior are consistently enforced (50%), that student 
misbehavior does not interfere with learning (40%), and that teachers, parents and 
administrators assume joint responsibility for student discipline (60%). Almost 100 % of 
respondents agreed to these items in the previous year. 

Despite the decline in mean scores from the initial to second year of operation, 
the mean ratings for MASE are higher than the national averages on each dimension, 
including Order. As indicated above, the overall score of 4.15 obtained for MASE can 
be contrasted with the overall score of 3.73 for secondary schools nationally. 

The qualitative responses further suggest a positive school climate and also 
provide insight about the slight decline in SCI ratings from year one to two. The 
principal described the climate as “realistic optimism” as opposed to last year’s “total 
optimism.” This year they can see “the bumps in road” but are still strongly committed 
to the school’s program and potential. High expectations were a dominant theme that 
emerged from the teacher interview data. “Here everyone is accountable. Everyone 
gets continual feedback, is expected to get results, and works collaboratively to address 
weaknesses.” The teachers appreciated the school’s leadership. They respect the 
principal and view him as a mentor who is supportive and understands their needs. The 
problem with student discipline reflected in the SCI ratings was echoed in some teacher 
comments. Discipline was described as difficult and students were said to be “quite 
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smart at breaking the rules.” High expectations were a major theme in student 
responses. The students talked about the hard work and long hours but these 
comments were tempered with recognition that teachers cared and wanted them to 
succeed. “Teachers are different here. Here they care and you must be respectful.” 

Smithson-Craighead Academy. The mean SCI ratings suggest that the school 
climate is average to favorable at this charter school. The overall mean rating is 3.94 
with dimension scores ranging from a low of 3.23 for Order to a high of 4.25 for 
Expectations. There was a consistent but mostly modest decline in mean scores on six 
of the seven dimensions and in the overall score when compared to last year’s scores. 
The mean score for Instruction remained constant over time. The largest decrease in 
mean scores was found for Order (from 3.86 to 3.23). An inspection of the individual 
items comprising this scale revealed that only about 16% of respondents agreed that 
student misbehavior does not interfere with teaching. 

When compared to the elementary school national SCI norms, SCA scores were 
largely comparable. In fact, the overall mean scores were nearly identical (3.94 for SCA 
versus 3.93 nationally). SCA had notably higher scores on the Expectations dimension 
when contrasted with the national average. On the other hand, the national average for 
Order was notably higher than that observed for SCA (3.50 nationally versus 3.23 for 
SCA). This finding is not supportive of the SCA goals for students of self-control, work 
before play, and obeying the teachers. 

The qualitative responses shed light on some of the trends observed in the 
quantitative ratings. The decrease in the Order mean to a level below the national norm 
may be due to the school’s large population of students with behavioral problems or 
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other kinds of special needs. According to the teachers, as many as 75% of the 
students are “medicated,” yet parents do not consistently fill prescriptions or administer 
the medications. One teacher suggested that SCA be noted as a special education 
charter school due to the academic and behavioral problems of the children. The goals 
for student behavior have become a central focus in the school. “In April, the principal 
began reading the behavior rules everyday.” More positive aspects of the school 
climate were apparent in the interview responses as well. For example, the principal 
described the climate as “safe and nurturing” and characterized teachers as supportive 
of the program and “willing to go the extra mile” to ensure its success. She has seen 
improvements in student self-esteem as well as academic performance. Students felt as 
if their teachers treated them with respect, which was not always the case in their 
previous schools. 

City University School of Liberal Arts. The mean ratings obtained on the SCI 
indicate a favorable school climate at this new charter school. This characteristic was 
evidenced by the overall mean rating of 4.10. The highest ratings were found for the 
Involvement and Instruction dimensions (both 4.29). Unanimous agreement on three of 
the items related to parental involvement contributed to the high mean rating on the 
Involvement scale. On the Instruction scale there was 100% agreement on two of the 
individual items: teachers design learning activities to support the curriculum and 

students needs, and pull out programs do not interfere with basic skills instruction. The 
lowest mean rating was scored on the Environment dimension (3.88). 

The mean school climate ratings for CityU also compared favorably to the 
normative sample of secondary schools. CityU means were higher than the national 
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averages on all dimensions and in a couple of areas this difference was large. The 
largest difference was observed on the Involvement dimension (4.29 for Cityll versus 
3.63 nationally). High rates of parent involvement are not common at public secondary 
schools. Respondents at Cityll also endorsed items on the Expectations dimension at 
higher rates when compared to national averages (4.20 for Cityll versus 3.82 
nationally). This finding was not surprising because high expectations are a defining 
characteristic of Cityll. Students are referred to as scholars and teachers as 
professors. 

Many of the interview responses also reflected a positive school climate. The 
principal stated that “the greatest accomplishment for Cityll is that parents, 
stakeholders, etc. are buying into the school.” The principal also described teachers as 
involved, supportive, and committed to the school vision. Teachers concurred, saying, 
“all teachers support the school’s mission and educational program.” Some students 
praised their teachers’ high expectations for their learning and appreciated the more 
individual attention afforded at a smaller school. These types of comments support the 
quantitative results suggesting that the Involvement, Instruction, and Expectations 
domains are strengths at the school. 

Star Academy. Star Academy had very high ratings on each dimension 
assessed by the SCI. These findings suggest that school climate is a strength at this 
charter school. The mean ratings ranged from a nearly perfect rating of 4.90 for 
Leadership to a low of 4.50 for Involvement. The overall score was an impressive 4.69. 

These mean ratings stand out even more when compared to national norms. 
Star Academy ratings were higher than the norms on each of the seven dimensions, 
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and many of these mean differences were large. The difference in Order scores was 
substantial (4.69 for Star versus 3.93 nationally). An examination of the results by 
individual item on the Order dimension reveals total or near total agreement on each 
item. 

The positive school climate indicated by the SCI ratings was supported by the 
qualitative data. The principal cited the biggest accomplishment as a “positive school 
climate conducive to learning.” Teachers were described as collaborative, supportive, 
familial, and even “happy.” These descriptors emerged even though it was clear that 
teachers were working very hard. One teacher noted, that “you don’t see anyone killing 
time here. This is the first school where the students and the teachers are on task every 
minute of the day. That has a lot to do with the mission of the school.” An appreciation 
for the schools’ leadership was also apparent in the teachers’ responses. They 
indicated that they were supported and treated professionally, and contributed to major 
decisions about the curriculum and program. Parents were characterized as supportive 
and involved, especially at the beginning of the year. 

Yo! Academy. SCI ratings at Yo! Academy were somewhat varied yet reflected a 
positive school climate overall (3.93). The highest mean rating of 4.20 was obtained for 
the Instruction dimension. All respondents agreed that teachers used a variety of 
teaching strategies, and designed learning activities to support curriculum and students’ 
needs. The lowest mean ratings were observed for the Involvement and Order 
dimensions (3.60 on both dimensions). Lower levels of agreement on items related to 
the active involvement of parents and community businesses in Yo! Academy 
contributed to the lower average rating for Involvement. On items comprising the Order 
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scale, more than half of the respondents indicated that tardiness or absence of students 
was a problem. Some respondents also indicated that student discipline was not 
administered consistently or appropriately. 

However, the average SCI ratings for Yo! were higher than the national norms for 
secondary schools on all but the Involvement dimension, on which the means were 
virtually identical (3.60 for Yo! and 3.63 nationally). Notably, the overall mean rating of 
3.93 was higher than the overall average of 3.73 nationally. 

The interview responses from the principal, teachers, and students were largely 
favorable with respect to school climate. The principal noted that faculty “enjoy the 
academic freedom and flexibility for aligning the curriculum.” Faculty concurred, saying 
that the “unique structure of the charter school gives teachers more autonomy over 
what they teach because the curriculum is not dictated.” The theme of high 
expectations for students was also present throughout the interviews. The “students as 
scholars” philosophy exemplified the high expectations and contributed to the favorable 
school climate. A familial atmosphere of mutual caring and respect was also evident in 
the qualitative data. Students talked about teachers who cared about their learning, 
provided individual attention, and understood their needs. To shed light on the 
somewhat lower ratings on the Order dimension, the principal explained that student 
absenteeism and “dissonance” was due to the difference between their year round 
school calendar and that of Memphis City Schools. The teachers talked about discipline 
problems as minor infractions, and students noted that they felt more comfortable and 
safe at Yo! than at their previous schools. 



50 




Summary findings across schools. Table 7 depicts the summary results based 
on SCI ratings and qualitative data collected for each of the seven charter schools. 
Overall, school climate is a clear strength of these charter schools. All schools were 
considered to be strong in school climate except SCA, which was rated as more 
moderate. This rating was based on the relatively low quantitative ratings on the Order 
dimension, which were inferior to national norms and decreased from Year 1 to Year 2, 
and associated qualitative responses. The interview responses confirmed this rating 
and suggest that student behavior and discipline problems, seemingly due in part to the 
highly at-risk student population enrolled, are adversely affecting school climate at this 
charter school. 

With regard to trends across time, it should be noted that three of the four Cohort 
1 schools showed decreases in SCI ratings. However, the overall mean ratings were 
still high in both in an absolute (above 4.00) and relative sense (higher than national 
norms). This trend of a gradual decrease in scores after the initial year is not 
necessarily surprising. As one principal so aptly argued, the staff moved from “total 
optimism” to “realistic optimism.” 
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Table 7 

Summary Results for School Climate 



Indicator 



School 



COSLA 

MAHS 

MASE 

SCA 

CityU* 

Star* 

Yo!* 



School Climate 
Ratings 



Qualitative 

Responses 




*First year schools 



Low 



Moderate/Average 




Strong 



Question 4. To what degree and levels of quality are the goals and strategies of 
the charter school being implemented? 

Data obtained from implementation benchmarks, the teacher questionnaire, 
teacher focus groups, principal interview, and classroom observations (SOM and 
RSCA) were used to address this evaluation question. To facilitate summarization of 
findings, the goals and strategies were divided into five major categories: curriculum, 
instruction, organization (leadership, governance), support (resources and professional 
development), and evaluation (accountability, assessment). See table 8 for summary 
findings. 
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Circles of Success Learning Academy. Substantial progress has been made in 
the achievement of benchmark goals at COSLA. The consensus of the leadership team 
was that all benchmarks were in the third and final phase of development. In most 
areas, the evidence gathered from other sources points to the achievement of these 
Phase III goals. However, evidence from these other sources also suggests that some 
benchmark indicators point more accurately to the second, intermediate phase of 
development. In the area of curriculum, COSLA continues to make progress in its 
implementation of the Success for All Program (SFA) in reading and math. According to 
the principal, the implementation of Science Exploration, and a fine arts program 
provided by the University of Memphis are in more preliminary stages of development. 
SOM and RSCA data indicate that instructional benchmarks may not be fully 
implemented. Cooperative learning, higher-order strategies, and teacher acting as a 
coach or facilitator were commonly observed, and these strategies are well aligned with 
the curricula and programs. Yet the percentage of time spent in direct instruction 
increased, and other student-centered strategies, most notably the use of technology, 
were rarely observed. In the remaining three categories of organization, support, and 
evaluation, notable progress was apparent and all data sources would indicate more 
advanced stages of goal implementation. Based on teacher perceptions, one exception 
to this trend may be the lack of support from state and local educational agencies. 

Memphis Academy of Health Sciences. In contrast to last year, when the 
implementation of benchmark goals was in the initial phases of development, there has 
been notable progress toward full implementation of several goals. The consensus of 
the leadership team was that benchmarks and indicators related to evaluation and 
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assessment were in Phase III of development. The teacher questionnaire ratings 
pertaining to regular review of goals to evaluate progress was endorsed by over 90% of 
respondents. Assessment also pertains to assessing student performance, and MAHS 
has emphasized the development and use of various student assessments aligned with 
state standards. 

Although the leadership team placed benchmarks and indicators related to 
curriculum and instruction at Phase III, data from other sources were somewhat 
contradictory. According to the open-ended responses from teachers and the principal, 
planning is still underway for adopting technology-based programs in math and science, 
and interdisciplinary, project-based learning. With regard to goals related to 
instructional strategies, SOM and RSCA data indicate that direct instruction and 
independent seatwork were most frequently observed and there was limited use of 
student-centered strategies. However, the incidence of student-centered did increase 
compared to the previous year. 

According to the benchmark document, the implementation of goals directed at 
organization and support were either Phase II or approaching Phase III. Teacher 
responses shed some light on the implementation of benchmarks in these areas. Their 
responses indicated a lack of resources and support. For example, there were low 
levels of agreement on items pertaining to adequate resources for technology, planning 
time, professional development, special education, and educational materials. Low 
levels of endorsement were also observed on support from parents, local or state 
educational agencies, and other external partners. Teachers were more positive on 
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items pertaining to organization and leadership. Nearly all indicated that they 
understood the school mission and were adequately involved in school decision making. 

Memphis Academy of Science and Engineering. Improvements were made not 
only in achieving the strategic goals but also in the benchmark document itself. 
Benchmarks in the areas of curriculum, instruction, and organization were developed to 
better reflect essential goals and strategies at MASE. The leadership team judged most 
benchmark components to be in Phase II. Although the SOM and RSCA data indicated 
an increase in student-centered activities from the previous year, these were not 
consistently observed and occurred at rates lower than those reported in national 
samples. In contrast, technology use did increase to levels that exceeded national 
norms. 

Other exceptions to the leadership’s Phase II designation for benchmarks were 
observed for indicators related to curriculum. The leadership team judged the 
integration of subject areas at 9 th grade, the elimination of remedial classes, and 
teachers’ use of a variety of resources to augment the texts to be in the initial Phase I 
stage of development. 

The only other indicator at Phase I pertained to common planning time, 
categorized by the leadership team as an organizational benchmark. Teacher 
responses suggest that lack of other resources, not planning time, may hamper 
progress toward full implementation of benchmarks related to support or organization. 
Whereas the large majority of teachers agreed that they were given sufficient planning 
time, there were lower levels of agreement about the adequacy of resources available 
for faculty and staff, professional development, and in particular, special education. 
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Because there were no benchmarks specifically pertaining to evaluation, it is 
more difficult to track progress in this area. However, nearly all teachers agreed that 
goals were reviewed regularly to evaluate progress. The revision of the benchmarks 
themselves is indicative of progress in this area, and the leadership team might 
consider adding benchmarks related to evaluation, assessment, and accountability in 
the next version of the document. 

Smithson-Craighead Academy. The majority of benchmarks are in the initial 
stages of implementation. The inclusion of new benchmarks, the introduction of new 
programs and strategies, and the hiring of new faculty account for the Phase I 
designation for many of these benchmarks. Two of the benchmark indicators were 
considered to be in Phase III. The first was the adoption of the Reading Success 
curriculum as a proven and scientifically-based program, and its alignment with state 
standards and assessment of student needs. The principal describes this program as 
a success, evidenced by improvements in student reading performance and motivation. 
The teachers also describe this program as successful for phonics instruction or in the 
lowest grade levels, but not as effective for reading comprehension or in higher grade 
levels. They also noted improvements in students’ reading skills and motivation. The 
other areas of curriculum are not fully implemented, and teachers agree that there is a 
lack of continuity because curriculum varies by individual teacher. This inconsistency 
may be why other benchmarks related to curriculum, including those pertaining to 
Reading Success, were considered to be Phase I or II. 

The second benchmark indicator deemed to be in Phase III was related to 
organization. More specifically, the indictor states that the leadership team functions as 
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the instructional leaders with the instructional staff, and that instructional staff have 
complete knowledge of the program and can articulate program needs. The teacher 
questionnaire data do not entirely support the achievement of this benchmark as it 
relates to shared decision-making. That is, only 20% of respondents agreed that 
teachers are adequately involved in school decision-making. On the other hand, nearly 
all teachers agreed that they had a thorough understanding of the school’s mission. A 
very similar benchmark indicator related to the leadership team was found under the 
support criterion, and was rated to be in Phase II. This suggests that the administrative 
team and staff meet regularly to discuss the needs of the school. It may be that the 
team meets regularly, but that teachers are not actively involved in the decision-making. 
As already noted, the consensus of the leadership team was that the remaining 
benchmarks in other categories were at the first stage of implementation, and data 
available from additional sources support this self-assessment. 

City University School of the Liberal Arts. The benchmark document indicates 
that all criteria were in the third and final phase of development. This is surprising given 
that CityU has completed only one year of operation. Part of the explanation for why 
they seemed to have made such quick progress rests with the benchmarks themselves. 
Some of the indicators are difficult to measure or objectively assess. For example, the 
indicator for Phase III under external support and assistance states that “all decisions 
made by personnel at CityU are in the best interest or our scholars and programs.” The 
evidence for this indicator includes list of recommendations and feedback from the 
president, professors, parents, and scholars. Thus, some benchmark indicators might 
be revised and better aligned with more objective evidence and written with more 
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distinct gradations across Phases, culminating in more challenging long-term goals. In 
fact, the indicator related to measurable goals and benchmarks specifies that the 
benchmark document will be updated on a regular basis. Although this benchmark 
indicator might be better addressed at an earlier phase, it should facilitate progress in 
evaluation and program improvement. 

Other benchmark indicators were easier to evaluate and could be informed by 
additional sources to help determine whether goals and strategies described in the 
benchmark document have been achieved at more advanced levels. The indicator for 
the instructional benchmark specifies the use of challenging, research-based teaching 
strategies for the majority of the instructional day in all core classes. Higher-level 
feedback and questioning were observed at rates comparable to the national sample, 
but the use of student-centered strategies, such as technology use or project-based 
learning, was not observed. Direct instruction was used the majority of the time. In 
terms of curriculum, the principal contends that the curriculum programs are fully 
implemented and aligned with state standards. In contrast, teachers suggested a need 
to improve the curriculum. 

Teacher survey responses illustrate goal achievement as it relates to 
organization and support. Three-fourths of the teachers did agree that they were 
adequately involved in school decision-making, and commented that they were “actively 
involved in the day-to-day running of the school.” Survey responses indicated a lack of 
support from the state and external partners as well as inadequate resources for 
educational materials, technology, and special education. Evidence for parental support 
was reflected in high ratings on related questionnaire items and positive comments from 
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teacher and principal interviews. In general, while several benchmarks do appear 
deserving of the Phase III ratings awarded by the leadership team, others appear to be 
at more developing levels. A more conservative orientation in future evaluations should 
be beneficial in establishing true attainments and areas of need. 

Star Academy. The consensus of the leadership team is that the implementation 
of goals and strategies are in Phases II and III of development. Indicators for the 
curriculum benchmarks were designated as Phase III, fully implemented, and aligned 
with state standards. The specific program identified for math and reading was 
McMillan McGraw Hill that entails integration across content areas and project-based 
learning. Though the principal judges the curriculum as fully implemented and 
successful, teachers seemed more ambivalent. Comments from the focus group 
suggest that progress in curriculum implementation still needs to made in order to be 
better aligned with TCAP. 

In the area of instruction, we would expect to see the strategies emphasized in 
the curriculum during classroom observations. Project-based learning was occasionally 
observed but interdisciplinary learning was rarely observed. Other student-centered 
and higher-order strategies were more commonly employed. 

The indicators for benchmarks related to organization and support were 
considered to be in Phase II, and survey data support this ranking. Teachers were very 
positive about the resources, support from most sources, and their involvement in 
decision-making. The exception was low levels of agreement to the item about 
assistance from external partners. Goals related to evaluation could be improved by 
revising the document to include benchmarks related to evaluation, assessment, and 
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accountability as well as more benchmarks in the category of support. All teachers did 
agree that goals were regularly reviewed to evaluate progress. 

Yo! Academy. Given that Yo! Academy is in its first year of operation, it is not 
surprising that most benchmark indicators were considered to be in the initial phase of 
development. The exceptions were goals pertaining to curriculum, instruction, and one 
area of support, which were designated as Phase II. However, the benchmark 
document was not very specific about what curricula or instructional strategies were 
targeted. The document identifies an “art infused, integrated curriculum” only under the 
Phase III indicator. The principal also describes the goals of integrating arts into the 
general curriculum and adopting specialty art classes. The teachers commented that 
there was no specific instructional model, but interdisciplinary “programs” or themes 
were implemented. SOM and RSCA data reflected little use of subject area integration 
and no project-based learning, which would be expected given the curricular focus. 
More traditional pedagogical strategies were most frequently observed. 

The benchmark for support, also judged to be at Phase II, was indicated by a 
common understanding and knowledge of the mission and goals of the school. The 
large majority of teachers agreed that they had a thorough understanding of the 
school’s mission and supported the school’s educational program. Evidence for initial 
stages of development for other goals and benchmarks were also reflected in teacher 
questionnaire responses. There were low levels of agreement on several items related 
to support and organization. Organization, especially leadership, was not favorably 
viewed by all teachers. Less than half of the teachers agreed that they were adequately 
involved in decision-making. In their open-ended responses, they further described a 
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lack of leadership and the “nepotism” that hampered effective decision-making and 
organization. As was the case with the other charter schools in their first year of 
operation, progress toward achieving goals related to evaluation might begin with 
improvement of the benchmark document itself. 

Summary findings across schools. Table 8 provides the summary ratings for the 
level of implementation of goals and strategies in five broad areas. Most schools were 
rated as moderate on their implementation of goals and strategies in the category of 
curriculum. COSLA seems to have made the most progress in this category. Not 
unexpectedly, two of the newest charter schools (Cityll and Yo!) were in very initial 
stages of curriculum implementation. Although some school leadership teams 
considered themselves to have made more progress in the implementation of 
benchmarks related to instruction, the data did not consistently support this designation. 
Again, all charter schools in their first year of operation were rated as “beginning” this 
area. SCA was also rated as “beginning” in the implementation of instructional goals 
due to contradictory evidence from the teachers, suggesting low levels of 
implementation and inconsistency across classrooms. 

Most charter schools were rated to be lower in the implementation of support 
goals. This rating was partially based on the school’s own designations and teacher 
questionnaire data suggesting a lack of support and resources in these schools. 
COSLA and Star were both rated as strong in this category, whereas CityLl received a 
moderate rating. The achievement of organizational goals was evaluated to be 
moderate in all but one school (Yo!). Finally, there was more variation in the ratings for 
the implementation of goals and strategies related to evaluation. COSLA and MAHS, 
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both second year schools, were considered to be strong in this category, whereas 
MASE and STAR received moderate ratings. The weak rating of the three other charter 
schools (two of them new) was partially due to lack of benchmarks in this area or the 
need to improve the benchmark document to identify and facilitate evaluation. 

Table 8 

Summary Ratings for Implementation of Goals and Strategies 

Indicator 

School 

COSLA 

MAHS 

MASE 

SCA 

CityU* 

Star* 

Yol* 

*First year schools 



Curriculum Instruction Support Organization Evaluation 




Beginning 



Moderate 




Strong 



Question 5. What are teacher reactions to and experiences in the charter 
school? What are the adequacy and quality of professional development and 
resources? 

The Charter School Teacher Questionnaire (CSTQ) and teacher focus group 
data were used to address teachers’ reactions to and experiences in the charter school, 
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including their responses targeting professional development and resources. The 
responses were organized into four categories. The first category (Program/Mission) 
encompassed two broad areas: teachers’ understanding of the mission and support of 
the educational program, and their perceptions that the program would result in 
successful student outcomes. The second category was professional development. 
The third category, resources, included whether teachers had adequate planning time, 
educational materials, technology, and a sufficient number of faculty and staff. The 
fourth and final category was support from the community, parents, and state and local 
educational agencies. See table 9 for summary findings. 

Circles of Success Learning Academy. COSLA teachers were unanimously 
positive in their ratings of indicators related to the school’s program and mission and 
their professional development. There was 100% agreement on items pertaining to 
these areas, and on most items there was an increase in percentages of agreement 
from the previous year. Teacher open-ended responses confirmed this finding. The 
only suggestion related to professional development was that new teacher orientation 
could be improved. Perceptions about the adequacy of resources were largely 
favorable and showed little change from the previous year. The only resource 
perceived to be inadequate was the ability to address the needs of children with special 
needs. In contrast to the ratings obtained in the previous year, a substantially higher 
percentage of teachers considered this problematic. There were no teacher comments 
that shed light on this trend. Perceptions of support from parents and external partners 
were evidenced by unanimous agreement with associated questionnaire items. Open- 
ended comments were also laudatory regarding parent support. Teacher perceptions of 
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support from the State Department and local educational agency or school district were 
more mixed. Although only half of the teachers agreed that they received effective 
support from these agencies, this percentage does represent an increase from the 
previous year. 

Memphis Academy of Health Sciences. The data reflected overwhelming 
teacher support for the school’s mission, educational programs, and beliefs that the 
program is linked to positive student outcomes. The ratings were consistently high 
across years. In contrast, positive ratings declined on the item pertaining to 
professional development, but the majority of respondents still agreed that professional 
development was adequate. One teacher commented that more staff development was 
needed. Teacher ratings further indicated the need for more resources for planning 
time, educational materials, technology, and special education. These perceptions 
were less favorable than in the previous year. The largest decline, from 100 to 55 
percent agreement, was observed for the availability of educational materials. Many of 
these needs were addressed in the open-ended comments, including the need for their 
own building and facilities. There were also notably less favorable perceptions of 
support from local and state educational agencies, external partners, and parents. For 
example, no teachers agreed that they received effective assistance from the local 
educational agencies; they tended to be neutral in their ratings of this item. Even 
though there was near unanimous agreement that parents had substantial opportunities 
to participate in their children’s education, few agreed that that they were active 
participants. The open-ended comments regarding support were sparse and somewhat 
mixed. 
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Memphis Academy of Science and Engineering. Teachers clearly support the 
mission and educational program of MASE. This enthusiasm for the mission and school 
was apparent not only in the questionnaire ratings but also in teacher interview 
responses. Even though more than half of teachers agreed that they had received 
adequate professional development, this was a decrease from nearly 90% in the 
previous year. The open-ended comments indicated that the long work hours made it 
difficult to engage in professional development opportunities during the academic year. 
The findings with respect to resources were mixed. Positive perceptions were indicated 
in the areas of planning time, educational materials, and technology. On the item 
concerning sufficiency of planning time, the percentage of respondents indicating 
agreement increased from 0 to 80 percent. In contrast, low and declining levels of 
agreement were observed for having a sufficient number of faculty and staff and 
adequately addressing requirements of special needs students. Teacher open-ended 
responses revealed that there was no specialist to work with special needs children and 
teacher turnover has been problematic because the number of staff is so small. More 
generally, the lack of support personnel and the insufficient number of administrators 
were noted. Teachers’ perceptions of support were also mixed. Support from parents 
and external partners was considered stronger than support from state and local 
education agencies. The percentage of teachers who agreed that they received 
effective assistance from the state did increase, whereas it remained constant for 
school district support. 

Smithson-Craighead Academy. The teacher questionnaire ratings reflected 
understanding of the mission, support for the program, and belief that the program 
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promotes student success. Even though there were high ratings on the item pertaining 
to professional development, the teacher focus group data suggested that professional 
development was inadequate at SCA. Teachers talked about the need for professional 
development related to behavior management due to discipline problems in their school. 
They contended that they were left “on their own” to find training, and it was not readily 
available from the district. High and increasing levels of agreement were observed for 
three areas of support-planning time, educational materials, and technology. Low and 
declining levels of support were shown for having a sufficient number of faculty and staff 
and addressing the requirement of special needs students. Problems with the behavior 
of special needs students and discipline were recurring themes in teachers’ open-ended 
comments. Perceptions of support were unfavorable in nearly all areas, including the 
assistance from the state and local educational agencies and the active participation of 
parents. There was more perceived support from external partners but this area still 
showed a substantial drop from the previous year. 

City University School of Liberal Arts. In the category of support for the program 
and mission, teachers’ reactions were very favorable. They concurred that they had a 
thorough understanding of the school’s mission, supported the educational program, 
and were confident that the program was linked to positive student outcomes. A pattern 
of mixed results was evident in the remaining categories. Although three-quarters of the 
teachers agreed that they received adequate professional development, the teacher 
focus group data indicated that it was all provided in-house or was more informal in 
nature. They were planning external professional development sessions, but had not 
received any training from the school district. Teachers suggested networking with 
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other teachers by establishing a consortium with other charter schools. Whereas there 
were high levels of agreement on items related to having adequate planning time and 
sufficient faculty and staff, there was less agreement that resources related to 
technology, educational materials, and special education were available. Teacher 
comments suggested that some educational and technological resources were provided 
later in the academic term. Two areas of support were perceived as strong - effective 
assistance from the local educational agency and active parental involvement and 
support. Rates of agreement were lower on two items reflecting effective assistance 
from external partners and from the State Department of Education. However, the 
teacher focus group data indicated that community support was “excellent”, suggesting 
some variation in perceptions of support among different types of external partners 
(e.g., community groups versus businesses). 

Star Academy. There was unanimous agreement on all items indicative of 
understanding the school’s mission, supporting the school’s educational program, and 
believing in the positive impact of the program on various student outcomes. Much of 
the open-ended data also reflected these themes. With respect to professional 
development, all teachers agreed that it was adequate to implement the educational 
program. In the focus group, teachers described a variety of professional development 
opportunities and the tone was positive. There were also high ratings observed across 
all categories of resources and support, with the exception of effective assistance from 
external partners. However, the open-ended comments were somewhat contradictory 
in these areas. Teachers identified community, business support, and quality parental 
involvement as areas in need of improvement. A recurring complaint was the lack of 
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support and resources from the school district. In addition, the teachers argued that 
there was a need for more space to accommodate their growing school. 

Yo! Academy. Teacher questionnaire ratings indicated positive teacher reactions 
in only the support for the mission and educational program category. On one 
component of this category, the program’s impact on student achievement was viewed 
more equivocally, with a little more than half of teachers endorsing this item. Teachers 
were more positive with respect to the program’s impact on students’ state assessment 
outcomes and encouraging high expectations for their work. A high regard for students 
and expectations for their success was confirmed in teachers’ open-ended comments. 
Less than 60% of teachers indicated that they received adequate professional 
development to support the educational program. Teachers described professional 
development opportunities as “pretty general” and not covering new information. The 
only resource that most teachers agreed to be sufficient was planning time. The other 
types of resources did not receive strong endorsements. Focus group responses 
revealed that laptop computers were “withdrawn” midyear and problems with instruction 
and textbook selection ensued. Others teacher said there were not enough textbooks, 
calculators, or overhead projectors. The lack of resources more generally was also 
addressed. Less than half of teachers agreed that assistance from any source was 
effective. The lowest ratings were obtained for the item concerning whether effective 
assistance was received by the local educational agency. However, in the focus group, 
community and parental support was favorably evaluated. Some open-ended 
comments pointed to the lack of support from school leadership and argued that it was 
inconsistent and unfair due to favoritism toward particular individuals or departments. 
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Summary findings across schools. The summary ratings indicate that teachers’ 
reactions were very positive with respect to the school mission or educational program 
(see Table 9). All charter schools were rated as strong in this category. There was 
more variation in the ratings for the professional development category. Teachers’ 
questionnaire ratings and open-ended comments both supported the strong rating for 
COSLA and Star. The moderate ratings for four of the charter schools (MAHS, MASE, 
SCA and Cityll) were based on the contradiction in findings between questionnaire 
ratings and open-ended comments. Yo! was the only school rated as weak in this 
category due to relatively low ratings and associated teacher comments. 

With respect to resources, COSLA was the only charter school to receive a 
strong rating in this category due to consistently high ratings across the four rating scale 
items targeting resources and favorable open-ended responses. Low ratings on 
resource items and less positive teacher comments accounted for weak ratings for 
MAHS and Yo!. The remaining charter schools were considered as moderate in this 
category due to inconsistent ratings on items comprising the support categories. No 
charter school was considered to be strong in the category of support. Moderate ratings 
(COSLA, MASE, CityU, and Star) resulted from high ratings on some questionnaire 
items but low ratings on others, or from conflicting trends noted in the open-ended data. 
Low ratings (MAHS, SCA, and Yo!) were based on the lower mean ratings across most 
support items augmented by less favorable open-ended comments. 
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Table 9 

Summary Ratings for Teachers’ Reactions 



Indicator 



School 



Program/ 

mission 



Professional 

development 



Resources 



Support 




*First year schools 



Low/Weak 



Moderate 




Strong 



Question 6. What are parent (caregiver) reactions to and experiences with the 
charter school? 

To address this final evaluation question, we relied primarily on quantitative and 
qualitative data collected on the parent questionnaire. It should be noted, however, that 
the return rate for parent questionnaires was relatively low, as normally occurs in 
schools serving low socioeconomic areas; thus the sample data may not be 
generalizable to the parent populations at the respective schools. By dividing the 
number of students enrolled by the number of parent respondents, we estimated that 
the response rates were 58% at COSLA, 48% at MAHS, 69% at MASE, 53% at SCA, 
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9% at CityU, 53% at Star, and 18% at Yo!. Clearly, the response rates at Cityll (n=1 1) 
and Yo! {n= 24) were extremely low, but further reflect the added difficulty of engaging 
parents of high school students in school-based activities. The questionnaire results 
were augmented by interview, benchmark, and observation data. For ease of 
expression, we refer to “parents,” but our use of this term also encompasses caregivers 
and guardians. The findings are organized into two categories: parent satisfaction and 
parent involvement. 

Circles of Success Learning Academy. Parent questionnaire data indicated 
continued satisfaction on nearly all indicators. These high rates of agreement were 
consistent across years. The only item that received a low rating concerned the 
adequacy of transportation services. On the open-ended questions, parent comments 
were overwhelmingly laudatory, confirming the rating scale data. They were particularly 
impressed with the principal’s vision and leadership and the teachers’ high expectations 
and caring for their children as individuals. They further appreciated the low student-to- 
teacher ratio and the positive, nurturing school climate. In terms of improvements, the 
suggestions pertained primarily to resources, especially the facilities. They wanted 
COSLA to have its own building and resources such as a physical education program, 
musical instruments, uniforms, technology, and the availability of more nutritious foods. 

Information gathered from various data sources all point to a high level of parent 
involvement as COSLA. Teachers unanimously agreed with the questionnaire item 
stating that parents were active partners with the school. In open-ended comments, 
teachers cited this involvement as one of the most successful aspects of their charter 
school. Parents are “required” to volunteer 20 hours of their time to the school. 
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Teachers commented that most fulfill this requirement and even exceed it. The 
benchmark data further indicates that 90 percent of parents have logged a minimum of 
20 volunteer hours. Although SOM data revealed parent or community involvement in 
learning activities rarely occurred, this finding may not be contradictory. That is, SOM 
only records strategy usage that is observed within the given classroom or its immediate 
vicinity. Parent involvement elsewhere in the school, such as on the grounds, the 
auditorium, or a parent room, would not be noted. 

Memphis Academy of Health Sciences. Consistently high ratings on nearly all 
questionnaire items indicate high levels of parent satisfaction at MAHS. When 
compared to the ratings observed in the previous year, the percentages of agreement 
were similarly favorable, which is impressive since nearly twice as many parents 
responded in the second year (86 versus 48). Some decreases in percentages of 
agreement were observed on items related to whether the physical surroundings were 
attractive and well maintained, yet the large majority of parents still endorsed these 
items. Only about 50% of parents agreed that transportation services are adequate, but 
this was still an increase from the previous year. Parents’ open-ended comments 
supported the high rates of satisfaction. When asked about the school’s strengths, 
most parents cited the academically challenging curriculum, the small student-to- 
teacher ratios, high expectations of students, principal leadership, and teacher quality. 
In response to suggestions for improvement, the most frequently occurring response 
was the need for their own school building that would provide more space to facilitate 
the adoption of physical education, sports, science labs, and performing arts. Parents 
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also thought that communication, discipline, and attitudes of some teachers could be 
improved. 

In contrast, the evidence suggests that parental involvement was not as high as 
parents satisfaction. Questionnaire data reveals a sharp decline in the percentage of 
teachers who agree that parents are active participants in the school (from 86 percent in 
the first year to 36 percent in the second). One teacher said that the participation of 
parents in the areas of academics and discipline needed improvement. Parental 
involvement was not specifically addressed in the principal or teacher focus groups, but 
students did mention that parents assisted with fund-raising efforts, communicated with 
the principal, and “helped out at school.” According to the benchmark document, 
parental involvement is at Phase II with indicators that emphasize support of school 
mission and goals as well as regularly held and well-attended parent meetings. More 
active parent involvement was indicated in the next phase of development. Parent or 
community involvement was not observed in or within the vicinity of the visited 
classrooms during SOM observations. 

Memphis Academy of Science and Engineering. Parents were very satisfied with 
the educational program at MASE. They were also satisfied with the school leadership, 
safety, discipline, and their own treatment in terms of being respected and encouraged 
to stay involved with the school. High rates of endorsement for these areas were 
consistent across years. There was some decline in the percentages of parents who 
agreed that they were regularly informed about their child’s progress and that teachers 
were readily available. They did not believe that transportation services were adequate 
in either year. Most parents cited the challenging curriculum, leadership, and teachers 
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as strengths of the school. Other themes included the high expectations for students, 
school safety, and discipline. In reference to areas in need of improvement, most 
responses centered on transportation and parking problems, the lack of communication 
between parents and teachers, school hours, and the need for extracurricular activities. 

Evidence from other sources suggests moderate rates of parental involvement. 
Almost three-fourths (70%) of teachers agree that parents are active partners with the 
school, but this figure represents a decline from almost 90% agreement in the previous 
year. There was also no evidence of parent involvement in the classroom (a rarity in 
high schools) based on SOM data. There were no benchmarks specific to parental 
involvement. According to both principal and teacher interview responses, parents have 
remained very involved in the school and, in fact, are more involved than they were last 
year. Students also described parent involvement as volunteers and counselors. Part 
of the explanation for their involvement appears to be the required hours of time 
specified in their contract agreement. Also, parent involvement in high schools 
compared to lower levels is more likely to be “behind the scenes” (e.g., fundraising, 
helping with special events) than in the classrooms or everyday instructional activities. 

Smithson-Craighead Academy. The questionnaire findings show that levels of 
parent satisfaction have remained consistently high across items and years. The 
percentages of agreement ranged from low of 72% on the desirability of charter versus 
regular schools to a high of 90% on whether they were treated respectfully and whether 
the curriculum was appropriate. Nearly all percentages of agreement were well above 
80%. The only notable decrease in scores was observed for the item on whether 
parents regularly receive information on their child’s progress. The dominant theme to 
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emerge from parent comments about the school’s strengths was their appreciation of 
the teachers. Other strengths noted were smaller class sizes, quality curricula, and an 
atmosphere of caring and concern for students. Responses to areas in need of 
improvement were varied. Many parents said “nothing” needed improvement. Others 
suggested more discipline, parent participation, and parent teacher conferences. It was 
not entirely clear whether the call for more parent participation referred to themselves or 
to other parents. 

As noted by the parents themselves, levels of parental involvement could be 
improved at SCA. The benchmark document indicates that the goal for community and 
parent involvement is in its initial phase. The teacher questionnaire data indicated only 
20% agreement that parents were active partners with the school. However, some 
community or parent participation was observed during classroom observations, but the 
principal’s interview responses suggested that community members were more likely 
than parents to participate in school activities. She stated that parental involvement 
“started out with a bang, then fizzled,” perhaps due to transportation problems. She 
further described future plans to increase involvement, including hiring a full-time after- 
school director dedicated to increasing parent participation. During the teacher focus 
group, one participant stated that parents do not feel comfortable coming to school due 
to difficulties at home. Students vaguely referenced parent participation by saying they 
“sometimes come to help with things” and “they tutor and stuff like that.” 

City University School of Liberal Arts. As previously noted, Cityll had an 
extremely low response rate, but the parents that did respond were very satisfied. 
There was unanimous agreement on several items, and 80-90 percent agreement on 
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others. The only two items with low agreement rates concerned the attractiveness of 
classrooms and hallways, and the adequacy of transportation services. In open-ended 
comments, parents cited the challenging academic program and the small class sizes 
that afforded individual student attention as strengths. The only recurring topics in 
parent suggestions for school improvement were to include extracurricular activities and 
strive for less teacher turnover. 

The evidence for parent involvement is somewhat contradictory, but on the whole 
suggests high levels of involvement, especially for a high school. The benchmark for 
parental involvement was rated as being at the third and final phase of development. 
SOM data did not indicate any community or parental participation during classroom 
observations, but such is not frequent in secondary education. The principal stated that 
there was a “slight decrease in parent” involvement from the first to the second 
semester due to fewer workshops offered. However, he still characterized parents as 
involved and supportive, partly due to their contractual obligation to attend at least four 
parent conferences annually. Teachers provided evidence to support parent 
involvement. Three-fourths of teachers agreed that parents were active partners with 
the school. In their focus group, teachers described parent involvement as strong 
though “not 100 percent.” Interviewed students commented that parents volunteered for 
a variety of activities and came to simply visit. 

Star Academy. Rating scale data indicated very high levels of satisfaction on all 
parent questionnaire items. The percentages of agreement ranged from 87% 
agreement for transportation adequacy to 100% agreement on several items. Parent 
open-ended comments were also very positive. Among the numerous strengths noted 
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were a challenging curriculum, low student-to-teacher ratios, teacher dedication, 
principal leadership, parental involvement, and a safe, welcoming atmosphere. 
Parents identified very few areas in need of improvement. The only recurring 
suggestion was to add more grade levels at Star. 

There is also evidence for moderate levels of parental involvement. The 
benchmark data places parental support and involvement at Phase II, specifying that 
about three-quarters of parents are volunteering and participating at the school. 
Ratings on the teacher questionnaire showed about 70% agreement with the item that 
parents are active partners with the school. The principal describes efforts to involve 
and inform parents but does not address the success of these efforts. Teachers, on the 
other hand, stated that parent participation “fizzled out” toward the end of the year. Yet 
they still consider parental attendance at programs as “good” and say that parental 
involvement at Star is much better than it was at their previous schools. Because 
students receive a grade for parental involvement, they may define it more broadly to 
include academic help. Several students commented that their parents helped them 
with their homework and to prepare for tests. SOM data indicated that parent or 
community involvement in the classroom was rarely observed. 

Yo! Academy. Very high rates of parental satisfaction were observed for Yo! 
Academy. The only item not as strongly endorsed was whether classrooms and 
hallways were attractive (67% agreement). The percentages of agreement for the 
remaining items ranged from 83 to 100 percent. The qualitative parent data supported 
these high ratings. Parents appreciated the low student-to-teacher ratios, the strong 
curriculum, teacher dedication, and a climate of high expectations for students. In 
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reference to areas in need of improvement, most parents suggested more parent 
involvement and communication, and more activities and classes. 

The leadership team considered the benchmark targeting parent and community 
involvement to be in Phase I, and other data confirm this designation. The SOM 
findings, like those for many secondary schools, did not reveal any parent or community 
participation in the classrooms. Teacher questionnaire data revealed that only 44% of 
respondents agreed that parents were active partners with the school. In the focus 
group, however, teachers said that parents were supportive and attended PTA 
meetings. They estimated that about 25% of the parents attended the last meeting. 
Teachers further described their efforts to inform and involve parents in school activities. 

The principal noted that more parental support was needed. She experienced 
some success promoting their involvement by scheduling parent meetings in 
conjunction with student performances. Students also said that parents could be more 
active in the school and suggested a parent-teacher support group. 

Summary findings across schools. Parent satisfaction was quite strong in all 
charter schools (See Table 10). Clearly, the parents had very positive reactions to 
these schools. Both parent rating scale data and their open-ended responses 
supported these findings. Ratings in the category of involvement were more varied. 
COSLA and Cityll were considered to be strong in this category because all or most 
data sources confirmed more advanced levels of involvement and rates of participation. 
The fact that both of these schools have parent contracts requiring their participation 
most likely accounts for these high levels of involvement. The moderate ratings for 
MASE and Star signaled contradictory evidence across data sources. The lower ratings 
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for MAHS, SCA, and Yo! resulted from consistently low indicators of parent participation 
and benchmarks suggesting achievement of this goal at more preliminary stages. 

Table 10 

Summary Results for Parental Satisfaction and Involvement 

School 
COSLA 
MAHS 
MASE 
SCA 
CityU* 

Star* 

Yo!* 

*First year school 



Satisfaction Involvement 




Beginning/Low 



Moderate 




Strong 
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Conclusions and Recommendations 



Conclusions by Evaluation Question 

The present study used a variety of data sources to address the evaluation 
questions guiding the second-year evaluation of Tennessee charter schools. The first 
cohort of four schools was initiated in the 2003-04 school year and has completed its 
second year of operation. The second cohort of three schools began operating in the 
2004-05 school year and has completed its first year. 

In response to the evaluation question that focused on instructional strategies, 
we found that traditional, direct instruction remained the dominant pedagogical practice 
across all charter schools. The rates were comparable to national norms in many 
cases, but still reflect limited success by the schools to implement the more motivating 
and progressive pedagogy described in benchmarks and instructional plans, or to take 
advantage of some of the unique conditions afforded by their charter status. That is, 
advantages of the charter schools cited by respondent groups and schools were smaller 
class sizes and more challenging instruction and curriculum. One might expect 
therefore to observe more student-centered, higher-order strategies implemented in 
these types of instructional environments. This expectation was reinforced by the 
schools’ own benchmarks in the areas of curriculum and instruction that comprised their 
school improvement plans. Although there were notable increases in these types of 
strategies for some second cohort schools, the rates were still low to moderate overall. 
RSCA data revealed that even when student-centered activities and technology usage 
was observed, their level of application was typically limited. Given the positive first- 
year achievement results (Ross et al., 2005), however, the traditional teaching methods 
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used appear to have been successful initially, but may not sustain positive effects as 
teaching staffs (perhaps with less talented presenters/lecturers than the original 
teachers) are expanded. 

The next evaluation question addressed school climate in the charter schools. 
As was reported last year, the findings reveal school climate to be a definite strength of 
these charter schools. Based on both quantitative and qualitative data, all but one 
(SCA) of the seven charter schools were rated as having a strong school climate. 
Further, on nearly all dimensions, the SCI ratings obtained for the charter schools were 
higher than national norms. Even though the results compared favorably to national 
norms, the decrease in SCI scores for three of the four schools in their second year of 
operation was noteworthy and should be monitored. Consistent decreases in scores 
were observed for the Order dimension in particular. Responses to indicators of Order 
were also less positive for Yol, a second cohort school. It appears that student 
discipline problems, absenteeism, and inconsistent rule enforcement may be adversely 
impacting climate at these schools. Nevertheless, Order represents only one of seven 
dimensions of school climate. Overall, the evidence points to a very favorable climate in 
the charter schools. 

Another question addressed the degree and level of quality with which the goals 
and strategies of the charter school were implemented. Although there was variation in 
the extent to which the goals and strategies were implemented, some patterns were 
detected. Not unexpectedly, the levels of benchmark implementation were more 
advanced among schools in their second year of operation compared to schools in their 
first year. However, the levels of implementation across most indicators were either the 
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first or second phase of development for all schools and years. Individual differences 
among schools notwithstanding, implementation tended to be moderate to strong in the 
areas of curriculum, organization, and evaluation. Implementation was weaker in the 
instruction and support areas. Yet another trend was for the school leadership teams to 
overestimate the extent of progress they had made in the implementation of benchmark 
goals and strategies. Given that schools are in their initial years of operation, we would 
not expect all goals and strategies to be fully implemented. In many cases, this 
tendency to overestimate progress signals a need to revise the benchmark document 
and become more comfortable with formative evaluation as a means for accurately 
assessing status and guiding improvement planning. 

In response to the question addressing teacher reactions and experiences with 
charter schools, we found that teachers were overwhelmingly positive about the 
school’s mission and program, but were less enthusiastic about available resources and 
support. Among first cohort schools, teacher ratings were lower in the areas of 
resources and support than they had been in the previous year. Responses in all but 
one (Star) of the seven schools suggested deficiencies in meeting the requirements of 
special needs students. It may be reasonably argued that this limitation is also related to 
organization or support, but it was often coupled with the perception of not having 
sufficient faculty and staff. In a smaller number of schools, resources (e.g., educational 
materials, technology) were perceived to be insufficient. In general, teachers were less 
likely to agree that they received adequate assistance from state and local educational 
agencies, and in some cases external partners. Evidence for adequate professional 
development was moderate to strong in all but one school (Yo!). 
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The final evaluation question focused on parent reactions to the charter school, 
and they were very favorable. Parent satisfaction is another definite strength of the 
charter schools. The areas for which they expressed less satisfaction were often those 
related to the physical building, facilities, surroundings, transportation, and lack of 
additional extracurricular activities such as sports programs. These limitations are more 
difficult for schools to address because they require substantial resources. In contrast 
to the uniformly high rates of parental satisfaction, levels of parental involvement were 
more modest and varied across schools. In some schools, parent involvement was 
considered to be parent support or communication rather than active, consistent 
participation in the educational program and more academically-related activities. 
Surprisingly, the level and extent of parent participation did not seem to differ depending 
on whether the school was in its first or second year of operation. The distribution of 
strong, moderate, and weak ratings was similar for first and second cohort schools. 

These conclusions should be tempered with the recognition of the limitations 
associated with the evaluation data. Small sample sizes and low response rates 
diminish the generalizability of findings. However, across the two years, the SOM 
results tended to be fairly consistent, and showed some progress for schools in targeted 
areas (e.g., classroom technology usage at MASE). When making comparisons 
between years for the first cohort schools, the fact that there were different groups of 
respondents may have contributed to the changing trends across time. Finally, we 
based some results on the benchmark documents, which varied in quality and were 
prepared by the leadership team, not the evaluators. Every effort was made to base the 
findings on multiple data sources. 
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Recommendations 



Based on the overall findings, the following recommendations apply to the 
charter schools as a group and not necessarily to individual schools. Due to a 
consistent pattern of findings across years, the recommendations are similar to those 
presented in the first year report. The recommendations specific to each school can be 
found in the individual school reports. 

First, charter schools could more closely align instructional practices with their 
school improvement plans and incorporate more research-based practices linked with 
student achievement outcomes. Many of the research-based practices call for the 
implementation of more student-centered strategies that entail higher-order thinking and 
technology integration. That is not to say that direct instruction has no place in 
classrooms, but rather a better balance among a variety of effective pedagogical 
practices might be achieved. The small classroom environment, viewed as a strength 
of charter schools in this study, would seem to be especially conducive for striking this 
balance. 

Second, even though school climate was a clear strength of the charter schools, 
order and discipline could be improved at some schools. School leadership teams may 
want to re-evaluate policies related to student conduct, discipline procedures, and 
attendance policies. Benchmark indicators targeting order might be developed to better 
formalize strategies to improve this dimension of climate. The findings on lack of 
resources to address the requirements of special needs students may have negatively 
impacted perceptions of order. Perhaps, better addressing the needs of these students 
would also improve school climate as it relates to an orderly environment. Another 
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strategy might include a closer partnership with parents in supporting appropriate 
behaviors and rules of conduct. 

Third, benchmark documents need to be modified to better align with objective 
indicators and available data. Some improvement in attaining benchmarks was noted 
among first cohort schools, and the leadership teams should be applauded for their 
efforts. Benchmark documents are intended to be a dynamic tool used for school 
improvement. Planning, evaluation, and ultimately improvement can be facilitated by a 
sound, ambitious, and long-range benchmark document. Results from this report, 
independent classroom observations, and student outcome data exemplify some of the 
evidence that might be provided to document goal attainment. 

Fourth, schools should make improvements in meeting the requirements of 
special needs students. As noted earlier, not meeting the needs of these students may 
contribute to the less orderly climate characterizing some schools. Additional resources 
might be earmarked to hire extra staff and faculty with expertise in special education. 
Policies and procedures related to addressing these needs might be developed or 
reviewed. 

Efforts to continue developing supportive, collaborative relationship with the 
school district and external partners should continue. The data indicate some progress 
in this area but the educational reform literature clearly emphasizes the importance to 
the sustainability of school change of having close connections to the parent school 
district. Strategies include having more open channels of communication and holding 
meetings to clarify policies, procedures, and available resources. Furthermore, 
representatives of charter schools should continue to seek the support and assistance 
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of various external partners. Publicity and notices in the form of newsletters, invitations 
to school functions, and community events might be used in conjunction with direct 
requests for assistance to promote more involvement from businesses, community 
groups, colleges, and parents. 

The final recommendation is to increase active parent involvement or 
participation. Schools that had the highest levels of parent involvement also had formal 
contracts of agreement with parents specifying the minimum number of hours of 
participation required. Furthermore, the definition and types of parental involvement 
might be reconsidered. In some school improvement plans, involvement translates into 
support for and understanding of the mission and educational program. In others, it 
specifies more active participation in parent organizations, school activities, and the 
provision of academic support for students. Both are necessary and important, and 
both might be included on benchmark documents that identify more specific strategies 
used to encourage involvement coupled with more objective evidence to indicate their 
effectiveness. 
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Appendix A 

School Observation Measure (SOM) Data Summary 

National Norms for Elementary Schools 



Number of School Observation Visits for N = 688 

Note: One school observation visit equals approximately 10 individual classroom visits. 



The extent to which each of the 
following was used or present in the 
school... 


Percent 

None 


Percent 

Rarely 


Percent 

Occasionally 


Percent 

Frequently 




Percent 

Extensively 


Instructional Orientation 


Direct instruction (lecture) 


3.0 


10.4 


23.7 


34.8 


28.1 


Team teaching 


49.8 


32.3 


11.9 


5.6 


0.4 


Cooperative/collaborative learning 


38.1 


41.8 


13.1 


5.2 


1.9 


Individual tutoring (teacher, peer, 
aide, adult volunteer) 


44.6 


31.8 


16.1 


5.2 


2.2 


Classroom Organization 


Ability groups 


36.3 


22.5 


15.7 


12.7 


12.7 


Multi-age grouping 


70.1 


15.7 


7.5 


4.1 


2.6 


Work centers (for individuals or 
groups) 


27.3 


27.7 


26.2 


15.4 


3.4 


Instructional Strategies 


Higher-level instructional feedback 
(written or verbal) to enhance student 
learning 


28.0 


36.2 


26.5 


7.1 


2.2 


Integration of subject areas 
(interdisciplinary /thematic units) 


58.3 


30.7 


7.6 


3.0 


0.4 


Project-based learning 


75.4 


15.8 


5.0 


3.8 


0.0 


Use of higher-level questioning 
strategies 


19.5 


32.7 


32.0 


12.4 


3.4 


Teacher acting as a coach/facilitator 


7.1 


20.2 


28.5 


34.5 


9.7 


Parent/community involvement in 
learning activities 


75.3 


18.4 


6.0 


0.4 


0.0 


Student Activities 


Independent seatwork (self-paced 
worksheets, individual assignments) 


4.1 


13.7 


39.6 


31.1 


11.5 


Experiential, hands-on learning 


30.5 


36.4 


24.9 


6.3 


1.9 


Systematic individual instruction 
(differential assignments geared to 
individual needs) 


64.9 


26.5 


5.6 


2.2 


0.7 


Sustained writing/composition (self- 
selected or teacher-generated topics) 


56.0 


32.8 


7.8 


2.2 


1.1 


Sustained reading 


43.3 


31.9 


18.5 


4.1 


2.2 


Independent inquiry/research on the 
part of students 


82.8 


13.1 


2.6 


1.5 


0.0 


Student discussion 


46.0 


24.5 


16.6 


9.1 


3.8 
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School Observation Measure (SOM) Data Summary, continued 
National Norms for Elementary Schools 



The extent to which each of the 
following was used or present in the 
school... 


Percent 

None 


Percent 

Rarely 


Percent 

Occasionally 


Percent 

Frequently 




Percent 

Extensively 


Technology Use 


Computer for instructional delivery 
(e.g. CAI, drill & practice) 


35.2 


37.1 


20.6 


6.7 


0.4 


Technology as a learning tool or 
resource (e.g., Internet research, 
spreadsheet or database creation, 
multi-media, CD Rom, Laser disk) 


49.6 


29.9 


16.4 


3.4 


0.7 


Assessment 


Performance assessment strategies 


66.4 


16.4 


9.9 


5.7 


1.5 


Student self-assessment (portfolios, 
individual record books) 


72.9 


18.7 


5.7 


2.7 


0.0 



Summary Items 


High academically focused class time 


0.4 


1.5 


17.1 


38.3 


42.8 


High level of student 
attention/interest/engagement 


0.4 


3.0 


20.1 


48.3 


28.3 





Rubric for SOM Scoring 


(0) None: 


Strategy was never observed. 


(1) Rarely: 


Observed in only one or two classes. Receives isolated use and/or little time in classes. 
Clearly not a prevalent/emphasized component of teaching and learning across classes. 


(2) Occasionally: 


Observed in some classes. Receives minimal or modest time or emphasis in classes. 
Not a prevalent/emphasized component of teaching and learning across classes. 


(3) Frequently: 


Observed in many but not all classes. Receives substantive time or emphasis in classes. 
A prevalent component of teaching and learning across classes. 


(4) Extensively: 


Observed in most or all classes. Receives substantive time and/or emphasis in classes. 
A highly prevalent component of teaching and learning across classes. 
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Appendix A, continued 

School Observation Measure (SOM) Data Summary 



National Norms for Secondary Schools 



Number of School Observation Visits for N = 370 

Note: One school observation visit equals approximately 10 individual classroom visits. 



The extent to which each of the 
following was used or present in the 
school... 


Percent 

None 


Percent 

Rarely 


Percent 

Occasionally 


Percent 

Frequently 




Percent 

Extensively 


Instructional Orientation 


Direct instruction (lecture) 


1.6 


8.7 


19.0 


44.0 


26.6 


Team teaching 


64.1 


30.3 


5.4 


0.3 


0.0 


Cooperative/collaborative learning 


35.7 


36.5 


23.2 


4.4 


0.3 


Individual tutoring (teacher, peer, 
aide, adult volunteer) 


76.5 


17.8 


1.9 


3.0 


0.8 


Classroom Organization 


Ability groups 


40.1 


29.8 


13.3 


7.3 


9.5 


Multi-age grouping 


53.7 


10.6 


8.9 


17.1 


9.8 


Work centers (for individuals or 
groups) 


69.4 


23.0 


7.3 


0.3 


0.0 


Instructional Strategies 


Higher-level instructional feedback 
(written or verbal) to enhance student 
learning 


25.1 


24.9 


25.7 


18.1 


6.2 


Integration of subject areas 
(interdisciplinary /thematic units) 


66.1 


21.7 


8.9 


1.4 


1.9 


Project-based learning 


51.1 


35.1 


10.6 


2.4 


0.8 


Use of higher-level questioning 
strategies 


19.2 


29.0 


32.8 


17.1 


1.9 


Teacher acting as a coach/facilitator 


3.8 


16.3 


27.9 


28.5 


23.6 


Parent/community involvement in 
learning activities 


93.0 


6.0 


1.1 


0.0 


0.0 


Student Activities 


Independent seatwork (self-paced 
worksheets, individual assignments) 


1.1 


9.8 


26.0 


39.6 


23.6 


Experiential, hands-on learning 


44.0 


36.4 


14.7 


4.6 


0.3 


Systematic individual instruction 
(differential assignments geared to 
individual needs) 


76.1 


20.9 


2.7 


0.3 


0.0 


Sustained writing/composition (self- 
selected or teacher-generated topics) 


54.5 


33.2 


10.4 


1.6 


0.3 


Sustained reading 


52.5 


36.0 


10.7 


0.8 


0.0 


Independent inquiry/research on the 
part of students 


67.7 


24.5 


7.1 


0.8 


0.0 


Student discussion 


65.7 


17.2 


8.7 


4.6 


3.8 
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School Observation Measure (SOM) Data Summary, continued 
National Norms for Secondary Schools 



The extent to which each of the 
following was used or present in the 
school... 


Percent 

None 


Percent 

Rarely 


Percent 

Occasionally 


Percent 

Frequently 




Percent 

Extensively 


Technology Use 


Computer for instructional delivery 
(e.g. CAI, drill & practice) 


49.6 


40.1 


8.1 


2.2 


0.0 


Technology as a learning tool or 
resource (e.g., Internet research, 
spreadsheet or database creation, 
multi-media, CD Rom, Laser disk) 


45.5 


37.9 


11.9 


4.6 


0.0 


Assessment 


Performance assessment strategies 


59.2 


26.3 


11.8 


2.7 


0.0 


Student self-assessment (portfolios, 
individual record books) 


77.7 


17.1 


4.1 


1.1 


0.0 



Summary Items 


High academically focused class time 


0.0 


1.6 


23.4 


59.4 


15.5 


High level of student 
attention/interest/engagement 


0.0 


3.3 


42.0 


46.9 


7.9 





Rubric for SOM Scoring 


(0) None: 


Strategy was never observed. 


(1) Rarely: 


Observed in only one or two classes. Receives isolated use and/or little time in classes. 
Clearly not a prevalent/emphasized component of teaching and learning across classes. 


(2) Occasionally: 


Observed in some classes. Receives minimal or modest time or emphasis in classes. 
Not a prevalent/emphasized component of teaching and learning across classes. 


(3) Frequently: 


Observed in many but not all classes. Receives substantive time or emphasis in classes. 
A prevalent component of teaching and learning across classes. 


(4) Extensively: 


Observed in most or all classes. Receives substantive time and/or emphasis in classes. 
A highly prevalent component of teaching and learning across classes. 
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